Standing service flaky-failures-report-from-ci-history-every-month · revised 12 October 2026
Standing service
Know every month which red builds were flaky and which tests cost the most
Each month the failed CI runs are sorted into flaky, real, infrastructure or configuration, and the ten tests that wasted the most minutes are ranked. Nothing in your repository changes.
This starts a conversation by email. Nothing is charged, and nothing is read or run, until we have agreed scope, access and terms with you in writing.
The responsibility you hand over
Flaky tests come back. Without a regular count, a team notices only when "just rerun it" is the habit again, and by then the list of causes is long and tangled.
Who it’s for: An engineering manager who wants a monthly view of CI trust without reading logs.
Usually starts when: You have sorted the history once, or fixed the worst tests, and want to see whether new ones appear.
The result: Every month the failed runs are sorted by the same written rule, the ten tests that wasted the most minutes are ranked, and a trend shows whether the number of flaky failures and wasted minutes is moving.
What stays true, and what we do about it
No response-time guarantee is published for this new service. A target for sending each month's report is agreed in writing before it starts.
The service runs once a month after the month closes. It is not a pager and has no round-the-clock cover.
What must remain true
- Every failed run in the month is classed by the written rule and linked
- The ranking uses the same wasted-minutes method every month so the trend is comparable
What we watch
- Failed runs and logs for the month, read-only
- The rerun history showing which runs passed on retry
When something happens
Scroll the table sideways to read it all.
| When | What we do |
|---|---|
| A month closes. | We send the classified list, the ranking, the trend and a summary. |
| One test wastes more minutes than the agreed limit two months running. | We offer to fix it as the named flaky-test job. Linked job: Stabilise flaky tests, priced per test |
| The share of runs with no rerun evidence rises above the agreed level. | We say the classes are unclear for those runs and suggest how to keep reruns visible. |
We do on our own
- Read run history and logs
- Classify runs and rank tests
- Offer a fixed job as a next step
We ask you first
- Any change to the classification rule
- Wider access
- Starting any fix
We escalate to you when
- Logs show a secret
- A test fails in a way that looks like a real defect, not flakiness
- History access is lost
How you know it held. Each month a script counts runs in the export against rows in the list, and your reader can recount the totals.
How we keep it true
This service is never finished. Each month's summary shows whether what it watches stayed true, and it continues until you end it.
Find which of your red builds are flaky and which tests waste the most time Job Optional
Up to 500 failed runs from the last 90 days are sorted into flaky, real, infrastructure or configuration, and your ten worst flaky tests are ranked by wasted minutes. Nothing in your repo changes.
Bought separately, once
The one-off sort of 90 days of history, with repeat runs of the top three tests if the suite runs in a sandbox.
Stabilise flaky tests, up to five, priced per test, each one fixed or openly quarantined Job Optional
Up to five named tests that fail at random, priced per test from £395 for one test. Each is reproduced, its cause found, then fixed or, with your approval, quarantined. You get a pull request.
Bought separately, when needed
The job that fixes or openly quarantines named flaky tests.
What is included, and what is not
- The monthly list of classified runs
- The ranking and trend
- A short summary
Included
- One repository on GitHub Actions and the month's failed runs, up to 150 a month, from a read-only token
- The same classification rule each month, with a link per run
- A ranking of the ten tests by wasted minutes and a trend
- A short summary and, on a trigger, an offer to fix the top test
Not included
- Fixing, quarantining or deleting tests
- CI without an API or export
- Logs with secrets until redacted
- Pager-style alerts
How we know it’s done
Agreed with you before work starts. Each check produces evidence you keep.
Each month every failed run in the export has a class and a link and the count matches the export.
Evidence: The script count and the monthly list.
The ranking and wasted-minutes figures can be recomputed from the list by your reader.
Evidence: The list with durations and the ranking formula.
Each quarter your engineer checks 20 runs chosen by script with a fixed seed and agrees with at least 18 of the 20 classes (the true rate is then at least 71% with 95% confidence). A run that was a real failure but that we called flaky is a severe error: more than 1 of the 20 fails the quarter.
Evidence: The 20 runs, the seed and your engineer's verdicts.
No file or setting in your repository was changed by us.
Evidence: Your repository audit log.
Sign-off. You read each report and tell us whether the rule still suits you. A month counts as delivered when you have received the report.
If it fails. If history is missing or changed shape we say so and do not guess. If the service is not working for you, you can end it at the end of any month.
When it fits, and when we stop
It fits when
- GitHub Actions or another CI with an API and run history
- Read-only run and log access
- A named engineer who checks a sample each quarter
We stop and tell you if
- Logs hold secrets that are not redacted
- The history stops showing reruns
- The token loses access and is not restored
What could go wrong
The service reads history and reports. Nothing changes in your repository or CI. Ending the service changes nothing.
Scroll the table sideways to read it all.
| Risk | How we handle it |
|---|---|
| A month with few failures gives a noisy ranking. | The report shows counts and says when too few runs exist to rank. |
| Logs hold secrets. | Logs are redacted for secrets by you or by a scan before they leave your account; a log that still holds one is returned and the job stops. |
| Test output carries instructions meant for the model. | Run logs and test output are treated as data: the model gets the text and returns rows a script validates, with no tools, no network and no secrets. |
A separate automated reviewer checks the work before hand-over. Your named engineer checks a sample each quarter. No human supervisor is included unless your agreement names one, and at launch the work is largely automated.
Questions
Do you fix the tests it finds?
No, only on request as a separate job.
Do I need the one-off history sort first?
Not required; the first month builds the rule. The one-off sort adds 90 days of history and repeat runs if the suite runs in a sandbox.
Do you need write access?
No. Read-only run history and logs.
Also part of
Send an enquiry
Send us
- Your CI system and failed runs a month
- Who reads the monthly report
- No logs, code or logins in the first enquiry
Later, once you agree
- A read-only token scoped to one repository, or a monthly export
- The classification rule agreed in the first month
- The name of your reader
You keep the CI account and repository; we read run history and logs through a read-only token or export. Logs are redacted for secrets by you or by a scan before they leave your account; a log that still holds one is returned and the job stops. Our automated agents read the files you send in a sandbox; we make no "no training" or "no storage" promise here, and the agreement names the model provider, country, retention and training status before anything is shared.
Email fallback: open your mail app
If website submission is unavailable, review and send the fallback email yourself. An email fallback is not a website receipt. Or write to hello@syntheticindustry.ai with “flaky-failures-report-from-ci-history-every-month” as the subject.