Job agent-pull-request-review-drill-for-your-team · revised 12 October 2026
Find out whether your reviewers catch the usual AI-agent mistakes: a scored drill
Your reviewers each review ten synthetic pull requests with planted agent mistakes, in their own time. You get team-level scores by mistake type and a short guide to what was missed.
You might be seeing
- Reviewers approve quickly because the code looks tidy
- Nobody knows which agent mistakes slip through
No passwords, keys, card details or admin invites needed to start.
What usually happened
AI-written code is fluent, so review habits built on spotting sloppy code fail to catch skipped tests, loosened thresholds and subtly wrong edge cases. Teams often find out which mistakes they miss only after one ships. Training on this is hard to check, because attending a session proves nothing.
Who it’s for: An engineering lead whose reviewers now read many AI-written pull requests and who wants to know which kinds of mistake they miss.
Usually starts when: Agent-written pull requests are rising, a mistake reached production that a reviewer should have seen, or you are about to hand more review to junior developers.
The result: Up to eight reviewers each review ten synthetic pull requests that contain planted agent mistakes, using their normal tools. You receive team-level scores by mistake type and a short guide. Each reviewer gets their own result first, and the lead sees it only if that reviewer agrees; the drill measures these ten pull requests, not how well anyone reviews real work.
Check whether this job fits
Answer these without sending code, tickets or logins. Nothing is submitted unless you choose to contact us.
What you get
- The ten synthetic pull requests
- The answer key, released after the drill
- Team-level scores by mistake type, and each reviewer's own result
- The scoring script and the guide
Included
- Ten synthetic pull requests on a small invented repository in your main language, with mistakes of the kinds GitHub's article lists (skipped tests, loosened thresholds, weakened CI, duplicated helpers, wrong edge cases) plus invented package names
- Up to eight reviewers, working alone and in their own time
- An answer key fixed before the drill and kept from reviewers
- Scoring by script, by mistake type; each reviewer's own result goes to that reviewer first
- A guide to the missed kinds, with links to published guidance
Not included
- A live trainer or workshop unless your agreement names a person
- Scores used for hiring, pay, promotion or performance decisions
- Review of your real pull requests or code
- A claim that anyone is or is not a good reviewer
- Any change to your repositories
How we know it’s done
Agreed with you before work starts. Each check produces evidence you keep.
The answer key is fixed and dated before the drill starts, and each planted mistake is shown to be real by a failing test, a skipped test or a package name the registry does not list on the day.
Evidence: The dated key and, for each pull request, the run or check that proves the mistake.
On two sets of recorded answers, from reviewers who agree to share them with you or, if none agrees, answers we write ourselves, that you score by hand against the key, the script's scores match yours on every item.
Evidence: The hand scores and the script output side by side.
Your lead rates three sampled pull requests as plausible for what your agents produce.
Evidence: The lead's written ratings.
Every reviewer received their own result before the lead received the team summary, and no individual result went to the lead without that reviewer's agreement.
Evidence: The delivery log.
Sign-off. Your lead reads the team summary and the guide and signs off in writing. Payment follows sign-off.
If it fails. If the agreed checks do not pass, you do not pay for this fixed scope. We hand over what we found and agree whether to stop or re-quote; no surprise work.
When it fits, and when we stop
It fits when
- Up to eight reviewers who agree to take part and see their own result first
- A lead who agrees in writing that individual results are for learning only and are seen by the lead only with that reviewer's agreement
- A main language we can write synthetic pull requests in
We stop and tell you if
- The lead intends to use individual scores for hiring, pay or performance decisions
- Reviewers have not agreed to take part
- The language or framework cannot be reproduced in a small invented repository
What could go wrong
The drill uses an invented repository and touches nothing of yours. Delete the repository and the results when you wish; there is nothing to revert.
Scroll the table sideways to read it all.
| Risk | How we handle it |
|---|---|
| Scores are used to judge people. | The agreement states learning only. Each reviewer sees their own result first, the report to the lead is team-level, and an individual result goes to the lead only if that reviewer agrees. |
| Ten pull requests is a small test and says little about real review. | The report says it measures these ten, by mistake type, and not overall review skill. |
| A planted package name could later exist on a public registry. | Invented names are checked against the registry on the day and the guide says never to install them. |
A separate automated reviewer checks the work before hand-over. Your lead checks three sampled pull requests look like what your agents produce. No human supervisor is included unless your agreement names one, and at launch the work is largely automated.
How we deliver
We arrange the work and independent review, then show you the result against the agreed checks. You keep authority over your systems.
- Agree the language, the reviewers and how scores will be used, in writing
- Write ten synthetic pull requests and fix the answer key before anyone sees them
- Confirm each planted mistake is real by running the code or tests
- Reviewers review alone; answers are collected by form or comments
- Score by script; separate automated reviewer checks key and scores
- Hand over scores, the key, the guide and the scoring script
This is a one-off job, not emergency cover or a subscription. We confirm eligibility, price, start window and delivery date before you accept, and work starts only after agreed inputs, secure access and permissions are in place. Model usage up to a cap we state in writing is in the price, and an enquiry creates no charge or booking.
Need to keep it working?
To check agent pull requests automatically before a person reads them, the pre-review check is offered separately.
Ongoing work is separately scoped and quoted: no monitoring, response-time guarantee or automatic subscription is included in this job.
Mention the responsibility you need in your enquiry.
What you can check
This is a new service. We have not delivered this job for a client yet.
Other ways to get this done
- Run a drill yourself using GitHub's published list of agent mistakes. It takes a senior's time to write the pull requests. github.blog
- Pair-review a few real agent pull requests together and write down what each person missed.
Questions
Is this training?
It is a scored drill with a guide, not a live session. Nobody delivers a talk unless your agreement names a person.
Can we use the scores in staff reviews?
No. The agreement says learning only, and we stop if you plan to use individual scores for performance decisions.
Does a good score mean someone is a good reviewer?
No. It measures ten synthetic pull requests, by kind of mistake.
Send an enquiry
Send us
- Your main language, framework and how many reviewers would take part
- Which review tool and which coding agents you use, by name of tool
- No code, pull request text or logins in the first enquiry
Later, once you agree
- The names of the reviewers who take part
- Your written statement that individual scores are for learning only
- The review tool and settings to mirror, in plain words
You keep everything. The drill runs on an invented repository we create; it needs no access to your code, accounts or logins. Each reviewer sees their own result first; the lead sees the team-level scores, and an individual result only if that reviewer agrees.
Email fallback: open your mail app
If website submission is unavailable, review and send the fallback email yourself. An email fallback is not a website receipt. Or write to hello@syntheticindustry.ai with “agent-pull-request-review-drill-for-your-team” as the subject.