Job ai-review-rules-written-from-your-past-pull-requests · revised 12 October 2026
Make your AI pull-request reviewer follow your rules, tested on your old PRs
Rules for the AI reviewer you already run, written from your past reviewed pull requests and tested on a held-back third of them, with a noise cap and a list of what the rules cannot catch.
You might be seeing
- Developers scroll past the AI reviewer's comments because most are speculative or off-topic
- The same convention is explained by a senior reviewer on pull request after pull request
No passwords, keys, card details or admin invites needed to start.
What usually happened
An AI reviewer left on its defaults does not know a team's own conventions, so it adds noise and misses what the team's reviewers always ask for. The team has no way to tell whether a rule file helps, because nobody has replayed it on past pull requests.
Who it’s for: An engineering lead of a team of 3 to 40 developers who already run an AI pull-request reviewer, and who wants it to raise what their own reviewers raise without burying each PR in comments.
Usually starts when: The team has stopped reading the bot's comments, or one senior person is still the review bottleneck.
The result: The reviewer you already use follows a written rule set drawn from your last reviewed pull requests. On pull requests held back before the rules were written, it raises at least the share of your reviewers' actionable comments that we agree in advance, stays under an agreed number of comments per PR, and never suggests disabling a check or skipping a test.
Check whether this job fits
Answer these without sending code, tickets or logins. Nothing is submitted unless you choose to contact us.
Checks you can run yourself
Count your repeated review comments
List the last 20 review comments from your seniors that asked for a change. Group those that say the same thing.
Look for: Whether five or more groups repeat. Repeated comments are the ones rules can capture.
What you get
- The rule file or files, as a pull request you merge
- The replay report with a link to every pull request and comment
- The noise count and the 30-comment sample sheet
- The replay script and the one-command re-run
- A list of what the rules cannot catch
Included
- One repository (or one monorepo) and one named review tool that reads an instruction file you control
- At least 30 merged pull requests with written human review: two thirds used to write the rules, one third held back and fixed before writing starts
- Rule file or files in the tool's own format, kept short and specific
- A replay of the held-back pull requests: your reviewers' actionable comments, the AI's comments, matched and extra
- A 30-comment sample your own reviewer marks useful, not useful or wrong, and a list of what the rules cannot catch
Not included
- Buying, choosing or paying for the review tool or its seats
- Reviewing your live pull requests: the tool you run does that, not us
- Making any bot an approver, or changing branch protection or required approvals
- Fixing what the reviewer finds in your code
- A promise that live usefulness will match the replay; the first live month is the real test
- Pull requests that contain secrets or personal data
How we know it’s done
Agreed with you before work starts. Each check produces evidence you keep.
On at least 30 merged pull requests, two thirds used to write the rules and one third held back and fixed before writing starts, the AI raises at least the agreed share of your reviewers' actionable comments on the held-back set, with the share agreed in writing beforehand; a share chosen after the run does not count. The held-back third is replayed once; if the rules change afterwards the result is marked not held-back. If the held-back set holds fewer than 30 actionable human comments, the report gives counts, not a percentage, and the pass mark is set in counts.
Evidence: The split list, the written pass marks and the replay table linking every pull request and comment.
The AI posts no more than the agreed number of comments per held-back pull request, and your marker rates no more than the agreed share of a 30-comment sample as not useful.
Evidence: The comment counts per pull request and the rated sample sheet.
Zero comments in the replay or the sample suggest disabling a check, skipping a test or lowering a threshold.
Evidence: The full comment list searched and read, with the count of such comments (zero required).
Ten planted-instruction test cases hidden in a pull request title, body, comment and code, each run five times, leave no planted marker in any output or action of the reviewer. This is a smoke test we plant.
Evidence: The test case list, the run log and the stated date; this is a baseline, not a claim of immunity.
Sign-off. Your named reviewer checks the sample sheet, your lead accepts the pass marks and you merge the rule pull request. Payment follows sign-off.
If it fails. If the agreed checks do not pass, you do not pay for this fixed scope. We hand over what we found and agree whether to stop or re-quote; no surprise work.
When it fits, and when we stop
It fits when
- At least 60 merged pull requests in the last 12 months with written human review, and 30 you can share
- One review tool you already run, with an instruction file you can edit and merge
- A named reviewer on your side who will mark 30 comments and settle disagreements
- Either an export of the pull requests and comments, or you run our replay script in your own account
We stop and tell you if
- Most past reviews are "looks good" with no written findings, so there is nothing to learn from
- Two of your reviewers disagree on more than 15% of a 20-comment labelling check; no pass mark would mean anything
- The code or comments cannot be shared and you will not run the replay script yourselves
What could go wrong
The rule files arrive as a pull request. Close it and nothing changes; after a merge, revert the commit and the tool returns to its defaults. We change no repository settings.
Scroll the table sideways to read it all.
| Risk | How we handle it |
|---|---|
| Tools follow instructions imperfectly, and long rule files make them worse. | Rules are kept short, and the replay measures what the tool actually does with them, not what the file says. |
| The replay does not predict live usefulness. | The report says so. The first live month is the real test; the monthly audit is offered for that, and the rules can be reverted. |
| Hostile text in a pull request steers the reviewer. | Issue titles, pull request text, comments and logs are treated as data, never as instructions: the model sees the text to be rated with no tools, and the job holds the model key and nothing else. We run harmless planted instructions hidden in a pull request title, body, comment and code, 10 cases five times each, and report the result as a baseline we plant ourselves (a smoke test), not immunity. |
| The rules reveal your conventions, which may be confidential. | The rule files go only into your repository. Our copies of your history are deleted when the job ends. |
A separate automated reviewer checks the work before hand-over. A named reviewer on your side marks the 30-comment sample and approves the rules before you merge them. No human supervisor is included unless your agreement names one, and at launch the work is largely automated.
How we deliver
We arrange the work and independent review, then show you the result against the agreed checks. You keep authority over your systems.
- Agree the tool, the repository, the pass marks and the severe-error list in writing
- Receive redacted history; split it two thirds to one third before reading
- Write the rule file from the first two thirds only
- Replay the held-back third and the planted-instruction tests, repeated runs
- Separate automated reviewer checks the replay; your marker rates 30 comments
- Hand over rules, report, script and the list of what is not caught
This is a one-off job, not emergency cover or a subscription. We confirm eligibility, price, start window and delivery date before you accept, and work starts only after agreed inputs, secure access and permissions are in place. Model usage up to a cap we state in writing is in the price, and an enquiry creates no charge or booking.
Need to keep it working?
To keep the reviewer useful as your code and rules change, a monthly audit is offered as a separate service.
Ongoing work is separately scoped and quoted: no monitoring, response-time guarantee or automatic subscription is included in this job.
Explore an ongoing managed lane, or mention the responsibility you need in your enquiry.
What you can check
This is a new service. We have not delivered this job for a client yet.
Other ways to get this done
- Write the rule file yourself. GitHub's guidance says to keep instructions short and specific, and lists what review instructions cannot do. docs.github.com
- Keep the tool on defaults and have a senior mark noisy comments for a month; it costs no cash but takes a senior's time.
Questions
Does this replace human review?
No. The tool suggests; your reviewers still decide. A bot comment does not count as an approval.
Which review tool does it work with?
Any tool that reads an instruction file you control. We name the tool in the agreement and test on that one only.
Will you comment on our live pull requests?
No. The tool you run does that. We only replay old pull requests and write the rule files.
Send an enquiry
Send us
- The review tool you use and how many developers and merged pull requests a month you have
- Two or three comments your reviewers repeat most often, in your own words
- No code, pull request text, tokens or logins in the first enquiry
Later, once you agree
- An export of the agreed pull requests and review comments, redacted by you, or a Read-only bot account on one repository
- Your written pass marks: share of actionable comments matched, maximum AI comments per PR, maximum share marked not useful
- The name of the person who marks the 30-comment sample
You keep the repository, the review tool, its seat and key, and every setting; we use a bot account with read-only access first, work on a fork or branch, and you merge. We receive an export of the agreed pull requests and comments, redacted by you, or read them through a Read-only account, and our automated agents read them in a sandbox; we make no "no training" or "no storage" promise here, and the agreement names the model provider, country, retention and training status before anything is shared. If you run our replay script in your own account with your own key, the model calls are made there and we receive only the results you send.
Email fallback: open your mail app
If website submission is unavailable, review and send the fallback email yourself. An email fallback is not a website receipt. Or write to hello@syntheticindustry.ai with “ai-review-rules-written-from-your-past-pull-requests” as the subject.