Rate every sampled comment with three words
Take 30 comments, or all of them if fewer. Mark each useful, not useful or wrong, and keep a link to the comment.
Wrong is worse than not useful: it sends someone the wrong way. GitHub says its own reviewer can miss problems and write comments that are wrong.
- Useful: changed the code or taught something
- Not useful: true but ignorable
- Wrong: false or misleading
Hold pull requests back before you write rules
Write rules from two thirds of your reviewed pull requests. Keep the last third unseen until the rules are done, then replay them. Rules tested on the pull requests they were written from will look good by construction.
Keep rules short and specific
GitHub suggests about 1,000 lines at most per file and warns that quality may deteriorate beyond that. It says vague requests such as "be more accurate" add noise, and it cannot make the tool block a merge.
- Start with a short list of clear rules
- Test a rule change in the same pull request
Read a pass mark with its sample size
If all 30 sampled items pass, the true rate is at least 90% with 95% confidence, by our own exact calculation (0.05 to the power 1/30). If all 50 pass, at least 94% (0.05 to the power 1/50). A pass mark without a sample size says little.
Write the mark, the sample and the list of errors that fail the whole test before you look at results.
Sources and limits
- GitHub: customize Copilot code review Checked 2026-10-12.
- GitHub recommends keeping any single instruction file to about 1,000 lines, and warns quality may deteriorate beyond that.
- Instructions cannot make Copilot block a merge, change formatting or follow external links, and vague requests such as "be more accurate" add noise.
- Copilot reads instructions from the pull request's head branch, so changes can be tested in the same pull request.
- Rule of three (statistics) Checked 2026-10-12.
- If nothing goes wrong in n trials, the 95% upper bound on the failure rate is about 3/n, which is 10% for 30 trials and 6% for 50.
- GitHub: responsible use of Copilot code review Checked 2026-10-12.
- GitHub says Copilot may miss problems and can produce comments that are wrong, so it should supplement human review.