Synthetic Industry

Collection · updated 2026-10-12

Adding AI to an engineering team: the order to put checks in, and what each one proves

A path from containing the workflow, through tuning the AI reviewer and checking agent pull requests, to judging bot pull requests, sorting CI failures and the backlog, with a bounded job at each step.

1. Contain the ground before you add an agent

GitHub advises pinning actions to a commit SHA and giving the token the minimum permissions. Text from issues and pull requests can reach an agent, so the agent job should be read-only and hold no secrets except a restricted model key with a spend cap and a minimal-permission workflow token.

This step proves the workflow is tidy. It does not prove it is secure.

  • Pinned actions
  • Minimum token permissions
  • No untrusted text in a shell or prompt

2. Make the reviewer follow your rules, and test it on old pull requests

Pick the tool on your own pull requests first, if you have not already. Then write short rules from your past reviews and replay them on pull requests the tool has not seen.

A replay shows what the rules did on old work. The first live month is the real test.

  • Compare tools on the same 30 pull requests
  • Rules from two thirds, test on one third
  • A noise cap and a list of what is not caught

3. Check agent-written pull requests before a person reads them

GitHub lists the usual agent mistakes: removed or skipped tests, loosened thresholds, duplicated helpers and code that passes and is wrong. A fixed checklist can look for the first three on every pull request.

A clean result means none of the listed mistakes was found. It does not mean the change is good.

  • Pre-review check on marked pull requests
  • A check on every new package
  • A scored drill for your reviewers

4. Give each bot pull request and alert a verdict

GitHub's AI review skips dependency files, so a bot pull request that only changes those files gets no Copilot review. A verdict says what changed, whether your code uses it and what the lockfile adds.

For alerts, a verdict rests on a tool's output you can re-run, not on a model's opinion.

  • Merge, hold or decide, with evidence
  • Alerts decided, none dismissed by us

5. Find out which red builds are real

Sort failed runs into flaky, real, infrastructure and configuration by a written rule. Then explain the real ones to authors in plain words, and never advise skipping a check.

  • A classified run list
  • The ten worst tests by wasted minutes
  • A plain-words comment on failed pull requests

6. Sort the backlog and the tickets

Sort the backlog once into keep, duplicate, needs information and proposed stale, and close nothing without a person. Then rewrite the tickets you will start so a developer can begin without asking.

In a public thread on a large project, a contributor argued that closing issues by silence alone loses useful ones; maintainers disagreed and the proposal was closed as not planned.

  • One state per issue
  • Tickets with checked citations
  • Open bug reports reproduced or explained

7. Keep it going

Each check drifts. The monthly versions re-measure it, and one lane bundles five of them with up to two bounded items a month.

Nothing here merges, approves or dismisses for you.

  • Monthly services
  • The software team agent lane

Sources and limits

  • GitHub: security hardening for GitHub Actions Checked 2026-10-12.
    • GitHub recommends pinning actions to a commit SHA when a lockfile is not possible.
    • The GITHUB_TOKEN should be granted the minimum required permissions, with more given to individual jobs only as needed.
    • Untrusted input such as a pull request title should go through an intermediate environment variable, not straight into a script.
  • GitHub: risks and mitigations of the Copilot coding agent Checked 2026-10-12.
    • GitHub's own agent can push to a single branch only, and its workflows do not run until someone with write access approves them.
    • GitHub prevents the person who asked the agent to open a pull request from approving it.
    • GitHub filters hidden characters, such as text in an HTML comment, before passing user input to the agent.
  • GitHub blog: agent pull requests are everywhere, here is how to review them Checked 2026-10-12.
    • Agents may take CI shortcuts such as removing tests, skipping the lint step or adding "|| true" to test commands.
    • Agents often duplicate helpers that already exist, and the riskiest output is code that compiles, passes every test and is wrong.
    • GitHub says any change that weakens CI is a blocker.
  • Microsoft .NET blog: ten months with the Copilot coding agent in dotnet/runtime Checked 2026-10-12.
    • Of 878 agent pull requests, 535 were merged and 253 closed, a 67.9% success rate against 87.1% for Microsoft engineers, though the populations differ.
    • Success was highest on cleanup at 84.7% and lowest of the named task types, performance, at 54.5%.
    • The author says the bottleneck has moved to review.
    • After a firewall fix and build instructions, success on agent pull requests rose from 38.1% to 69%.