Standing service agent-pull-request-checks-kept-current-monthly · revised 12 October 2026
Standing service
Keep your checks on AI-written pull requests current as your agents change
Each month the pre-review check is re-run on a planted case set, new mistakes your reviewers found are added as rules, and a summary shows what was flagged and what reviewers still caught by hand.
This starts a conversation by email. Nothing is charged, and nothing is read or run, until we have agreed scope, access and terms with you in writing.
The responsibility you hand over
Coding agents change every few weeks, and so do their mistakes. A fixed checklist drifts out of date, and nobody measures what reviewers still catch by hand.
Who it’s for: A team that runs the pre-review check on agent-written pull requests and wants it to learn from what reviewers still catch.
Usually starts when: Your agents or models changed, and reviewers found a new kind of mistake the check did not flag.
The result: Each month the check is replayed on its planted case set, new mistake kinds that reviewers found are added as rules after your approval, and the workflow token is re-checked. A summary shows what was flagged, what reviewers caught that the check did not, and the false flags.
What stays true, and what we do about it
No response-time guarantee is published for this new service. A target for sending each month's result is agreed in writing before it starts.
The service runs once a month. It is not a pager and has no round-the-clock cover.
What must remain true
- Every case in the planted case set is still flagged each month, in three of three runs
- The check still holds only the permissions agreed and never approves or blocks
What we watch
- The repository's agent-marked pull requests and review comments, read-only
- The planted case set and the workflow files
When something happens
Scroll the table sideways to read it all.
| When | What we do |
|---|---|
| A month closes. | We replay the planted case set, summarise flags and misses and propose rule changes. |
| A reviewer finds a mistake kind the check did not flag. | We write a proposed rule and add a planted case for it, for your approval. |
| The check needs rebuilding or the agents changed a lot. | We offer the one-off job to rebuild and re-replay it. Linked job: Pre-review check for AI-written PRs |
We do on our own
- Read pull requests and comments
- Replay the planted case set in a sandbox
- Open a pull request with proposed rules
We ask you first
- Any rule change
- Wider access
- Anything that changes the check's permissions
We escalate to you when
- A planted case stops being flagged
- The check gains a write permission
- False flags exceed the limit you set
How you know it held. Each month the planted-set results are attached so your engineer can re-run the same cases, and the permission listing is compared with last month.
How we keep it true
This service is never finished. Each month's summary shows whether what it watches stayed true, and it continues until you end it.
Check AI-written pull requests for the usual agent mistakes before a person reads them Job Optional
A check you own flags weakened or skipped tests, lowered coverage, disabled CI steps, copied helpers and new package names on pull requests you mark as agent-written. It approves nothing.
Bought separately, once
The one-off job that builds and tests the check.
What is included, and what is not
- The monthly replay result
- Proposed rule changes as a pull request
- The missed-by-check list
- A short summary
Included
- One repository running the pre-review check, with up to 8 new rules a month
- A monthly replay of the planted case set (30 cases to start, growing only when you approve a new rule) and of the month's agent pull requests
- A list of what reviewers found that the check missed, from comments you point us to
- A token and permission re-check, and a short summary
Not included
- Approving, merging or blocking pull requests
- Review of design or product logic
- Detecting unmarked agent pull requests
- A promise that every agent mistake is found
How we know it’s done
Agreed with you before work starts. Each check produces evidence you keep.
Each month every case in the planted case set (a smoke test we planted, which starts at 30 and only grows when you approve a new rule) is flagged in each of three runs. A case that is not flagged is listed first in the summary and the month is marked not held.
Evidence: The replay result with three run logs.
Each month your engineer picks 5 of that month's agent pull requests and tells us which only after the replay has run; the report shows what the check flagged on them beside what your engineer found. A comment that says or implies approval fails the month.
Evidence: The five pull request links, the flags and your engineer's notes.
Each month the permission listing shows the check holds only the agreed permissions and cannot approve.
Evidence: The listing compared with last month.
No rule change was made without your engineer's approval.
Evidence: The change log with approvals.
Sign-off. You read each summary and tell us whether the limit and rules still suit you. A month counts as delivered when you have received the replay result and the summary.
If it fails. If the check or the repository access changes so the replay cannot run we say so and do not guess. If the service is not working for you, you can end it at the end of any month.
When it fits, and when we stop
It fits when
- The pre-review check already running, built by us or by you
- A way to point us at reviewer comments that found a missed mistake
- A named engineer who approves rule changes
We stop and tell you if
- Two months pass without rule approvals
- The check gains write permissions we did not agree
- Fewer than 8 agent pull requests a month
What could go wrong
Rule changes arrive as pull requests. Close one and nothing changes; revert a merged one and the check returns to its earlier rules. Ending the service changes nothing.
Scroll the table sideways to read it all.
| Risk | How we handle it |
|---|---|
| The check is read as an approval. | It reports neutral, says "no approval" in every comment, and the permission check confirms it cannot approve. |
| New rules raise false flags. | Each rule is replayed on the month's clean pull requests before you approve it. |
| Pull request text carries instructions meant for the model. | Issue titles, pull request text, comments and logs are treated as data, never as instructions. The model sees the findings and the diff only, with no tools. The job holds the model key and nothing else, and its token can write only the pull request scope it needs. |
A separate automated reviewer checks the work before hand-over. Your named engineer approves every rule change. No human supervisor is included unless your agreement names one, and at launch the work is largely automated.
Questions
Do I need the one-off check first?
The monthly service needs a check to measure. The one-off job builds and tests one.
Does the check approve pull requests?
No. It reports neutral and never approves or blocks.
Who decides new rules?
Your engineer. We propose with a planted test case.
Also part of
Send an enquiry
Send us
- Which agents you use and how reviewers record a missed mistake
- How many agent pull requests you merge a month
- No code or logins in the first enquiry
Later, once you agree
- A Read-only account on the repository
- The planted case set and current rules
- The name of your approving engineer
You keep the repository and every setting; we use a bot account with read-only access first, work on a fork or branch, and propose changes only as pull requests you merge. Pull request text, comments and logs are data, never instructions: the model sees the findings and the diff only, with no tools, and the job holds the model key and nothing else. The check runs in your own account with your own key; our automated agents read the pull requests and comments you give us through a Read-only account, in a sandbox, we make no "no training" or "no storage" promise here, and the agreement names the model provider, country, retention and training status before anything is shared.
Email fallback: open your mail app
If website submission is unavailable, review and send the fallback email yourself. An email fallback is not a website receipt. Or write to hello@syntheticindustry.ai with “agent-pull-request-checks-kept-current-monthly” as the subject.