Job golden-set-test-for-an-existing-ai-extraction-job · revised 12 October 2026
Make our AI extraction step fail a test when a change pushes it below an agreed floor
Your AI extraction or tagging step gets a 100-row labelled test set and a one-command test that fails when whole-record accuracy falls below an agreed floor or the output shape changes.
You might be seeing
- Output quality changed after a prompt edit or a provider update, and no test caught it
- Nobody has a labelled set to re-run when something changes
No passwords, keys, card details or admin invites needed to start.
What usually happened
A model step can work at launch and drift a month later, with no error to show for it. Structured output guarantees the shape of the result, not that the values are right. Without a labelled set and a test that fails on a regression, every change is a guess.
Who it’s for: Developer or data engineer with a live model step for invoices, tagging or categories, built in n8n, Make or Python.
Usually starts when: A prompt tweak or a provider model update changed the output and nobody noticed for weeks.
The result: Your step gets a 100-row labelled golden set and a one-command test that fails when whole-record accuracy drops below the agreed floor or the output shape changes. The test is shown to fail on 5 planted regressions and pass 3 harmless changes, and it never uses a model as the grader.
Check whether this job fits
Answer these without sending files or logins. Nothing is submitted unless you choose to contact us.
What you get
- The golden set
- The harness script and one-command run
- The thresholds file
- The planted-regression results
- Run steps with the cost per run
Included
- One live model step that produces structured output
- A golden set of 100 rows, labelled with you
- A harness that runs on your own provider key or on recorded outputs
- A thresholds file holding the whole-record floor, field floors and the shape check
- A proof run: 5 planted regressions and 3 benign changes, and the cost per run
Not included
- Running your production step or touching production data
- Holding your provider key
- Improving the prompt or model
- Ongoing monitoring, which is the monthly service
- A model as the grader
- Detecting a drop of a few points: 100 rows catch large drops, a changed output shape and flipped cases, not small average changes
How we know it’s done
Agreed with you before work starts. Each check produces evidence you keep.
The test fails on each of 5 planted regressions (a prompt edit dropping a field, a model swap, a date-format change, a sign flip, truncation) and passes 3 benign changes. Two of the five regressions and one of the three benign changes are made by you and kept from us until the test run; the other three regressions and two benign changes are ours, so they show the harness works, not that it catches every real regression.
Evidence: The planted-change table with each verdict.
Five live runs of your unchanged step on your own key give the same verdict, or the report shows the spread and sets the floor below it. If you run on recorded outputs only, this check is not run and the report says so.
Evidence: The five run logs, or the note that this check was not run.
Grading is by exact match and field checks in code; no model grades any output.
Evidence: The harness source and its imports.
Cost per run is stated and under the cap agreed in writing.
Evidence: The cost line in the run steps.
The shape check fails on a planted output with a missing field and on one with an extra field.
Evidence: The two planted outputs and the verdicts.
Sign-off. Your developer runs the harness, inspects the proof table and signs off in writing. Payment follows sign-off. The test covers the golden set; it is not proof about future inputs.
If it fails. If a check above is not met, you do not pay for this fixed scope. You keep the results and the failures, and we agree whether to stop or re-quote. No surprise work.
When it fits, and when we stop
It fits when
- The step produces structured output you can compare field by field
- 100 golden rows that are non-personal or redacted
- Two people on your side label the same 50 rows and agree on at least 43
- A developer who runs the harness with your own key
We stop and tell you if
- No ground truth can be set for the step
- The golden rows hold personal data that cannot be redacted
- You want access to production
- Two labellers agree on fewer than 43 of 50 rows
What could go wrong
The job reads copies you send and writes files. It changes nothing in your systems, so there is nothing to revert. Our copies of your files are deleted when the job ends.
Scroll the table sideways to read it all.
| Risk | How we handle it |
|---|---|
| The golden set goes stale as your inputs change. | The thresholds file and the golden set carry a review date, and the monthly service refreshes them. |
| The test fails for no reason, from model variance. | Where you run it on your own key, grading is exact-match and field checks. Five live runs of the unchanged step are compared, and the floor is set below the spread they show. |
| A model grades another model. | No model grades anything in this test. All grading is in code. |
A separate automated reviewer checks the harness, the floors and the proof run before hand-over. A developer on your side labels the golden rows with us, runs the harness and signs off. No person on our side is included unless your agreement names one, and at launch the work is largely automated.
How we deliver
We arrange the work and independent review, then show you the result against the agreed checks. You keep authority over your systems.
- Agree the step, the output fields, the floors and the severe errors
- Two of your people label 50 rows to check they agree
- Build the golden set with you and the harness with exact-match grading
- Plant 5 regressions and 3 benign changes; run five times each
- Independent review of the harness, the floors and the proof run
- Hand over the harness and golden set; delete our copies
This is a one-off job, not emergency cover or a subscription. We confirm eligibility, the price, a start window and a delivery date before you accept, and work starts only after inputs, a secure handoff route and permissions are in place. Model usage up to a cap agreed in writing is included, and an enquiry creates no charge or booking.
Need to keep it working?
Want the golden set re-run each month with fresh rows? The monthly drift check does that.
Ongoing work is separately scoped and quoted: no monitoring, response-time guarantee or automatic subscription is included in this job.
Explore an ongoing managed lane, or mention the responsibility you need in your enquiry.
What you can check
This is a new service. We have not delivered this job for a client yet.
Other ways to get this done
- Open-source evaluation tools such as promptfoo can run test sets in CI. They give you a runner; the labelled set, the floors and the planted-regression results are the work. www.promptfoo.dev
- Structured output settings from your provider fix the shape of the answer, not the correctness of its values. developers.openai.com
Questions
Why not let a model grade the output?
Models can get graded wrong too. Exact-match grading is repeatable, which is what a test needs.
Do you need my provider key?
No. You run the harness with your own key, or on recorded outputs.
Does this make my step more accurate?
No. It tells you when it gets worse.
Will it catch a small drop in accuracy?
Not reliably. 100 rows catch large drops, a changed output shape and cases that flip. A drop of a few points that stays above the floor can pass. Catching small drops needs a much larger labelled set.
Send an enquiry
Send us
- What the step does and its output fields, as text
- Which tool it runs in (n8n, Make, Python) and how often it runs
- No real records, personal data or credentials in the first enquiry
Later, once you agree
- 100 redacted or synthetic input rows and their expected outputs by an agreed secure route
- A way to run the harness with your own key, or recorded outputs
- A named developer who runs it
- A written cap on cost per run
You keep your step, your key and your data. We work on redacted rows you send by an agreed secure route, return a harness you run yourself, and delete our copies when the job ends. Your provider key never comes to us.
Email fallback: open your mail app
If website submission is unavailable, review and send the fallback email yourself. An email fallback is not a website receipt. Or write to hello@syntheticindustry.ai with “golden-set-test-for-an-existing-ai-extraction-job” as the subject.