The problem
A reviewer left on its defaults does not know your conventions. It adds comments nobody asked for and misses what your seniors always say.
In a public thread, one developer said it is hard to stop such a tool listing speculative reasons the code is a problem.
What a rule file can do
GitHub says to keep instruction files short, around 1,000 lines at most, and to give specific rules. It also lists what instructions cannot do: block a merge, change formatting or follow links.
So a rule file shapes what the tool says. It cannot make the tool an approver.
- Short and specific
- Tested on old pull requests
- Never an approval
What to ask for
Ask for a replay on pull requests held back before the rules were written. Ask for pass marks, a comment cap and a list of what the rules cannot catch, all agreed in writing first.
- A held-back third
- A cap on comments per pull request
- Zero comments that suggest skipping a check
Which job fits
If you have not picked a tool, compare up to three on the same 30 pull requests. If you have one, write the rules and replay them. If it is running and drifting, audit a sample of its comments each month.
Sources and limits
- GitHub: customize Copilot code review Checked 2026-10-12.
- GitHub recommends keeping any single instruction file to about 1,000 lines, and warns quality may deteriorate beyond that.
- Instructions cannot make Copilot block a merge, change formatting or follow external links, and vague requests such as "be more accurate" add noise.
- Copilot reads instructions from the pull request's head branch, so changes can be tested in the same pull request.
- GitHub: Copilot code review Checked 2026-10-12.
- By default Copilot's reviews do not count toward required approvals.
- Dependency management files such as package.json and Gemfile.lock are not reviewed.
- A review costs about $0.05 to $1 (Lite) or $0.25 to $5 (Balanced) in AI credits, not counting Actions minutes.
- Hacker News: comment in the thread "There is an AI code review bubble" Checked 2026-10-12.
- A commenter wrote on 2026-01-26 that it is "really hard to get it not to tell you 20 highly speculative reasons why the code is problematic along with the one critical error".