Job flaky-failures-sorted-from-ci-history · revised 12 October 2026
Find which of your red builds are flaky and which tests waste the most time
Up to 500 failed runs from the last 90 days are sorted into flaky, real, infrastructure or configuration, and your ten worst flaky tests are ranked by wasted minutes. Nothing in your repo changes.
You might be seeing
- The same build is rerun until it passes
- Nobody can name the five tests that fail most often
No passwords, keys, card details or admin invites needed to start.
What usually happened
When red builds are normal, real failures get merged over. A team cannot decide whether to fix, quarantine or delete flaky tests until someone has sorted the failed runs by cause and counted the minutes each test wastes.
Who it’s for: An engineering manager whose team reruns failed builds by habit instead of reading them.
Usually starts when: People say "just rerun it", and nobody can tell which red builds were real or which tests cost the most time.
The result: Up to 500 failed runs from the last 90 days each get a class and a link: flaky, real, infrastructure or configuration. The rule for each class is written down, and your ten worst flaky tests are ranked by wasted minutes with a likely cause type. The top three are repeat-run in a sandbox with the counts reported (repeat runs only if the suite runs in a sandbox), and nothing in your repository changes.
Check whether this job fits
Answer these without sending code, tickets or logins. Nothing is submitted unless you choose to contact us.
What you get
- The classified run list with links
- The classification rule in plain words
- The ranking of ten tests
- The wasted-minutes formula and the per-run durations
- The repeat-run counts for the top three, if the suite runs in a sandbox
Included
- One repository on GitHub Actions and up to 500 failed runs from the last 90 days, from a read-only token or an export
- A written classification rule, for example "failed then passed on the same commit"
- A ranked list of the ten worst flaky tests by wasted minutes, with a likely cause type labelled as inference
- Repeat runs of the top three tests in a sandbox, with fail counts
- Your check of 20 sampled runs
Not included
- Fixing, quarantining or deleting any test; that is a separate job
- CI systems without an API or an export we can read
- Runs older than 90 days or more than 500 runs unless quoted
- Logs that hold secrets, until redacted
- A promise that a cause type is correct; it is an inference unless the repeat run shows it
How we know it’s done
Agreed with you before work starts. Each check produces evidence you keep.
Every failed run in scope has one class and a link to its run, and the classification rule is written in the report.
Evidence: A script count of runs in the export against rows in the list.
Your engineer checks 20 runs chosen by script with a fixed seed and agrees with at least 18 of the 20 classes (the true rate is then at least 71% with 95% confidence). A run your engineer shows was a real failure but we called flaky is a severe error: more than 1 of the 20 fails the test.
Evidence: The 20 runs, the seed and the engineer's verdicts.
If the suite runs in a sandbox, the top three tests are repeat-run there five times and the pass and fail counts are reported.
Evidence: The repeat-run logs and counts.
No file, setting or workflow in your repository was changed by us.
Evidence: Your repository audit log for the period.
Sign-off. Your engineer reads the sample check and the ranking and signs off in writing. Payment follows sign-off.
If it fails. If the agreed checks do not pass, you do not pay for this fixed scope. We hand over what we found and agree whether to stop or re-quote; no surprise work.
When it fits, and when we stop
It fits when
- GitHub Actions, or another CI with an API, and read access to run logs through a read-only token or an export you make
- Run history that shows reruns and commits
- A named engineer who checks 20 sampled runs
We stop and tell you if
- Logs contain secrets that cannot be redacted
- The history does not show reruns, so flaky and real cannot be told apart
- The suite cannot run in a sandbox: we drop the repeat runs and the repeat-run check, say so in writing before work, and the price stays £195
What could go wrong
The job reads run history and writes a report; nothing changes in your repository or CI. Our copies of your logs are deleted when the job ends.
Scroll the table sideways to read it all.
| Risk | How we handle it |
|---|---|
| Missing reruns make a flaky failure look real. | The rule is written down, the report states how many runs had no rerun evidence, and those are marked unclear. |
| Logs hold secrets or personal data. | Logs are redacted for secrets by you or by a scan before they leave your account; a log that still holds one is returned and the job stops. |
| Test output carries instructions meant for the model. | Run logs and test output are treated as data: the model gets the text and returns rows a script validates, with no tools, no network and no secrets. |
A separate automated reviewer checks the work before hand-over. A named engineer on your side checks 20 sampled runs. No human supervisor is included unless your agreement names one, and at launch the work is largely automated.
How we deliver
We arrange the work and independent review, then show you the result against the agreed checks. You keep authority over your systems.
- Agree the repository, the 90-day window and the classification rule
- Receive redacted run history and logs read-only
- Class every failed run by rule; link each to its run
- Rank tests by wasted minutes; repeat-run the top three in a sandbox
- Separate automated reviewer checks the classes; your engineer checks 20 runs
- Hand over the list, rule, ranking and counts
This is a one-off job, not emergency cover or a subscription. We confirm eligibility, price, start window and delivery date before you accept, and work starts only after agreed inputs, secure access and permissions are in place. Model usage up to a cap we state in writing is in the price, and an enquiry creates no charge or booking.
Need to keep it working?
To fix the worst offenders, the flaky-test job follows. To keep the report coming, a monthly service is offered separately.
Ongoing work is separately scoped and quoted: no monitoring, response-time guarantee or automatic subscription is included in this job.
Explore an ongoing managed lane, or mention the responsibility you need in your enquiry.
What you can check
This is a new service. We have not delivered this job for a client yet.
Other ways to get this done
- Export failed-then-passed pairs from your CI history yourself and count the tests that recur.
- Some CI providers, such as CircleCI, show flaky test reports in their own dashboards; check yours first. circleci.com
Questions
Will you fix the flaky tests?
No. This report tells you which ones matter. Fixing the top ones is a separate job.
Is the cause type certain?
No. It is a labelled inference unless the repeat run shows it.
Do you need write access?
No. Read-only run history and logs are enough.
Send an enquiry
Send us
- Your CI system and roughly how many failed runs a month you see
- How long a full run takes
- No logs, code or logins in the first enquiry
Later, once you agree
- A read-only token for run history and logs scoped to one repository, or an export you make
- Read-only code at one commit for the repeat runs, through a company-controlled route
- The name of your checking engineer
You keep the CI account, the repository and every setting; we use a dedicated bot account with the least access that works (read-only first), with read-only run and log access. Logs are redacted for secrets by you or by a scan before they leave your account; a log that still holds one is returned and the job stops. Our automated agents read the files you send in a sandbox; we make no "no training" or "no storage" promise here, and the agreement names the model provider, country, retention and training status before anything is shared.
Email fallback: open your mail app
If website submission is unavailable, review and send the fallback email yourself. An email fallback is not a website receipt. Or write to hello@syntheticindustry.ai with “flaky-failures-sorted-from-ci-history” as the subject.