Job agent-written-pull-requests-checked-before-human-review · revised 12 October 2026
Check AI-written pull requests for the usual agent mistakes before a person reads them
A check you own flags weakened or skipped tests, lowered coverage, disabled CI steps, copied helpers and new package names on pull requests you mark as agent-written. It approves nothing.
You might be seeing
- A pull request passes CI because a test was skipped or a threshold was lowered
- Reviewers spend their time on mechanical problems before they reach the design
No passwords, keys, card details or admin invites needed to start.
What usually happened
Coding agents make a recognisable set of mistakes: tests skipped or dropped, coverage or thresholds loosened, helpers created twice, edge cases subtly wrong, new dependencies nobody chose. Reviewers find them by hand, one pull request at a time, and a plain AI review comment does not check for them on purpose.
Who it’s for: A lead whose team ships pull requests written by coding agents, and whose reviewers are now the bottleneck.
Usually starts when: Agent-written pull requests arrive faster than people can read them, and reviewers keep finding the same kinds of mistake.
The result: Every pull request you mark as agent-written gets one short pre-review comment from a check you own, listing the usual agent mistakes found, from a fixed list. It never approves, never merges and never sets a required status. A planted-defect replay shows which kinds it flags.
Check whether this job fits
Answer these without sending code, tickets or logins. Nothing is submitted unless you choose to contact us.
What you get
- The check workflow and rules as a pull request you merge
- The checklist, in plain words
- The replay report with 10 planted defects, 5 held-back defects and 20 clean pull requests
- A frozen set of 30 cases and the one-command re-run
- A summary at day 30
Included
- One repository on GitHub Actions and one marker that says a pull request is agent-written (a label, a branch prefix or the author)
- A fixed list of checks: skipped or deleted tests, lowered coverage or thresholds, disabled CI steps or continue-on-error, duplicated helpers, new package names (flagged only by name against the lockfile before the change), and a missing failing-first test for bug fixes
- Rules first, in plain code; a language model only to write the short comment from what the rules found
- A planted-defect replay on copies of your past pull requests, plus 5 defects your engineer plants after the rules are frozen, and a 30-day window with one summary
Not included
- Approving, merging or blocking any pull request
- Reviewing the design or the product logic of the change
- Detecting agent-written code you have not marked
- Package age, maintainer and install-script checks; the new-dependencies job does those
- A promise that every agent mistake is found; only the listed kinds are checked
- Changes to branch protection, required checks or repository settings
How we know it’s done
Agreed with you before work starts. Each check produces evidence you keep.
In a replay of 10 defects we planted (a skipped test, a lowered threshold, continue-on-error, a duplicated helper, a new package name and five more agreed kinds), every defect is flagged in each of three runs; this part is a smoke test. The held-back part: 5 further defects of the same kinds, planted by your engineer after the rules are frozen and not shown to us, are all flagged in each of three runs. A comment that says or implies approval fails the whole test.
Evidence: The planted list and the three run logs side by side, and the frozen rule commit hash dated before your engineer's 5 defects. It describes these kinds, not every agent mistake.
On 20 clean past pull requests the check raises no more false flags than the number agreed in writing beforehand.
Evidence: The 20 results with a link per pull request and the agreed limit.
The workflow token holds contents: read plus exactly one write scope (pull-requests or checks), listed in the file. A push, approve or merge attempt in a sandbox is denied by the platform, and the replay log shows exactly one comment per pull request.
Evidence: The permissions block, the denied-action log and the comment count per pull request.
Ten planted-instruction test cases hidden in a pull request title, body, commit message and code comment, run five times each, leave no planted marker in any output. This is a smoke test we plant.
Evidence: The test case list and run log with the date; a baseline, not a claim of immunity.
Sign-off. Your named engineer reads the replay and the token list, and signs off in writing. You merge the pull request. Payment follows sign-off.
If it fails. If the agreed checks do not pass, you do not pay for this fixed scope. We hand over what we found and agree whether to stop or re-quote; no surprise work.
When it fits, and when we stop
It fits when
- GitHub Actions, and at least 8 agent-written pull requests a month
- A marker for agent-written pull requests that your team applies
- Your repository setting that lets workflows approve pull requests is off, confirmed by you
- 30 past pull requests we may copy for the replay
We stop and tell you if
- Pull requests are too large for the agreed size cap, or contain secrets
- The setting that lets workflows approve pull requests is on and you will not turn it off
- No past agent-written pull requests exist to build the replay from
What could go wrong
The check arrives as a pull request; close it and nothing changes. After a merge, delete the workflow file and the check stops. We change no repository settings.
Scroll the table sideways to read it all.
| Risk | How we handle it |
|---|---|
| The comment is read as an approval. | It says "no approval" in every comment, reports neutral, is never a required check, and the approve setting must be off. |
| Hostile text in the pull request title or body steers the model that writes the comment. | Issue titles, pull request text, comments and logs are treated as data, never as instructions. The model sees the findings and the diff only, with no tools. The job holds the model key and no other secret, and its token holds one write scope. |
| Agent pull requests are often partly rewritten by people, so the checklist misses new mistake types. | Only the listed kinds are checked, and the day-30 summary lists what reviewers found that the check did not. |
A separate automated reviewer checks the work before hand-over. Your named engineer approves the checklist and the false-flag limit before the workflow is merged. No human supervisor is included unless your agreement names one, and at launch the work is largely automated.
How we deliver
We arrange the work and independent review, then show you the result against the agreed checks. You keep authority over your systems.
- Agree the marker, the checklist, the size cap and the acceptable false flags
- Copy 30 past pull requests and plant 10 defects in some of them
- Build the rules check and the comment writer on a branch
- Replay planted and clean pull requests three times, then your engineer's 5 held-back defects
- Separate automated reviewer checks permissions, replay and planted-instruction tests
- Hand over the pull request, the report and the frozen case set
This is a one-off job, not emergency cover or a subscription. We confirm eligibility, price, start window and delivery date before you accept, and work starts only after agreed inputs, secure access and permissions are in place. Model usage up to a cap we state in writing is in the price, and an enquiry creates no charge or booking.
Need to keep it working?
To keep the checklist current as your agents change, a monthly service is offered separately.
Ongoing work is separately scoped and quoted: no monitoring, response-time guarantee or automatic subscription is included in this job.
Explore an ongoing managed lane, or mention the responsibility you need in your enquiry.
What you can check
This is a new service. We have not delivered this job for a client yet.
Other ways to get this done
- Use GitHub's own list of agent mistakes as a manual checklist in your pull request template. github.blog
- Ask your reviewers to apply the same checklist by hand; it costs no cash but costs reviewer time on every pull request.
Questions
Does it approve or block pull requests?
No. It leaves one short comment and a neutral status. People approve and merge.
Does it work on pull requests we have not marked?
No. It runs only on pull requests with your agent marker.
Is a clean check a sign the pull request is safe to merge?
No. It only says none of the listed kinds of mistake was found. Review is still needed.
Send an enquiry
Send us
- How many agent-written pull requests you merge a month and how you mark them
- Which coding agents your team uses, by name of tool
- No code, tokens or logins in the first enquiry
Later, once you agree
- 30 past pull requests, redacted by you, or a Read-only bot account on one repository
- The marker rule and the size cap in writing
- Your agreed number of acceptable false flags on 20 clean pull requests
You keep the repository, the workflow runner and every setting. The check runs in your Actions with a token that holds contents: read plus exactly one write scope (pull-requests or checks), listed in the file. The finished check runs in your own account with your own key; for the replay we receive 30 past pull requests you have redacted, or read them through a Read-only account, and our automated agents read them in a sandbox, we make no "no training" or "no storage" promise here, and the agreement names the model provider, country, retention and training status before anything is shared.
Email fallback: open your mail app
If website submission is unavailable, review and send the fallback email yourself. An email fallback is not a website receipt. Or write to hello@syntheticindustry.ai with “agent-written-pull-requests-checked-before-human-review” as the subject.