Standing service monthly-ai-job-drift-check-on-golden-set · revised 12 October 2026
Standing service
Every month, check our AI step still gets the same answers; flag a fall or shape change
Each month the outputs of your golden set and 20 fresh rows you label are scored against the agreed floor, with a report of any fall and the rows that changed. You send the outputs; no live access.
This starts a conversation by email. Nothing is charged, and nothing is run, until we have agreed scope, data route and terms with you in writing.
The responsibility you hand over
A model step can drift after a provider update, a prompt edit or a change in the inputs, with no error to show for it. A one-off test catches the changes you make. It does not catch the ones made for you, or the slow change in what arrives.
Who it’s for: Developer or data engineer whose live model step for extraction or tagging must not silently get worse.
Usually starts when: A provider update or a prompt edit changed the output once and went unnoticed for weeks.
The result: Every month you receive a report that scores the outputs of your golden-set run, plus 20 fresh rows from last month's live outputs that you label. It shows the scores against the floor, and the rows that changed. Any fall below the floor is raised in the report with the evidence.
What stays true, and what we do about it
No response-time guarantee is published for this new service. A date for each month's report is agreed in writing before it starts, set to what a service at this stage can keep.
Runs once a month after the sample arrives. At launch the service is not staffed round the clock.
What must remain true
- Each month the golden set and 20 fresh labelled rows are scored against the agreed floor
- Any fall below the floor, or any change in the output shape, is raised in that month's report with the rows that changed
What we watch
- The golden set and thresholds file
- 20 fresh rows from last month's live outputs that you label and send
- The golden-set outputs for the month, from your harness run
When something happens
Scroll the table sideways to read it all.
| When | What we do |
|---|---|
| A month closes and the 20 fresh labelled rows arrive. | We score the golden-set outputs and the fresh rows you send and return the scores with the changed rows. |
| A score falls below the floor. | We list the rows that changed and name a likely cause only where the evidence shows it. We offer a refreshed golden set. Linked job: Golden-set test for an AI step |
| The output shape changes. | We report the changed fields and stop scoring until you confirm the new shape. |
| The fresh rows show a kind of input the golden set lacks. | We propose rows to add to the golden set. You decide. |
We do on our own
- Score the outputs you send
- Prepare the scored report
- Offer a refreshed golden set
We ask you first
- Any change to the floor, the thresholds or the golden set
- Any request for wider access, for a login or for your provider key
We escalate to you when
- Two months in a row fall below the floor
- The output shape changes
- The sample you send is missing or changes shape
How you know it held. Each month you can recompute every score from the files we return, and the report states the number of rows scored and the number that changed.
How we keep it true
This service is never finished. Each month's report shows whether the thing it watches stayed true, and it continues until you end it.
Make our AI extraction step fail a test when a change pushes it below an agreed floor Job Optional
Your AI extraction or tagging step gets a 100-row labelled test set and a one-command test that fails when whole-record accuracy falls below an agreed floor or the output shape changes.
Bought separately; not part of the monthly fee
The one-off job that builds the golden set and harness whose outputs this service scores.
What is included, and what is not
- A monthly scored report
- The list of rows that changed since last month
- The updated golden-set history
- A short note on any fall and its evidence
Included
- One live model step with a golden set, from our golden-set job or your own
- A monthly scoring of the golden-set outputs you send (you run the harness, ours or yours, with your own key, or send recorded outputs)
- 20 fresh rows a month from last month's live outputs, labelled by you
- A scored report: floor, scores, changed rows and shape check
- A golden-set history, kept as the durable record
Not included
- Access to your production step, data or provider key
- Improving or retuning your prompt or model
- Fixing a regression: findings are reported, and any fix is a separate agreed job
- Out-of-hours cover or any guaranteed response time
- A model as the grader
How we know it’s done
Agreed with you before work starts. Each check produces evidence you keep.
Each month's report covers the golden set and the 20 fresh rows, and its scores can be recomputed from the files returned.
Evidence: The report, the files and your recomputation.
Any score below the floor is raised in the report with the rows that changed.
Evidence: The report's section on falls.
No live access was used: the sample came from you, and nothing was changed in your systems.
Evidence: Your own audit trail for the month, which shows no access by us.
The report states the rows scored and the rows that changed, and no model graded any output.
Evidence: The report header and the harness source.
Sign-off. You read each report and tell us whether the marks and rules still suit you. A month counts as delivered when you have received the report.
If it fails. If we cannot run a month because a sample is missing or has changed shape, we say so, explain what we need and do not guess. If the service is not working for you, you can end it at the end of any month.
When it fits, and when we stop
It fits when
- A golden set of at least 100 labelled rows exists, with a thresholds file
- A named person labels 20 fresh rows each month, about 15 minutes in all
- The rows are non-personal or redacted
- You run the harness, or send recorded outputs
We stop and tell you if
- No golden set exists and none can be built
- Fresh rows cannot be redacted
- You want us to hold your provider key or production access
- The output shape changes every month so scores cannot be compared
What could go wrong
The service reads samples and writes reports, so there is nothing to undo. Ending it leaves your systems exactly as they are; you keep every report we sent.
Scroll the table sideways to read it all.
| Risk | How we handle it |
|---|---|
| The golden set goes stale and the scores look better than reality. | Each report states the age of the golden set and proposes new rows from the fresh sample. |
| A fall is caused by variance, not a change. | Grading is exact match in code, and a fall is raised with the rows that changed so you can judge. |
| 20 fresh rows a month can catch only a large drift. | The report says so and counts the rows scored. A small average change can pass. |
| Your files go to a model provider outside your control. | Before any file is shared we name the provider, its country and its retention terms in writing. Entry jobs run on redacted, synthetic or public copies, or on your own provider account with a key you hold. |
A separate automated reviewer checks each month's scores, the changed rows and the shape check before hand-over and does not mark the monthly sample. A named person on your side labels the 20 fresh rows and signs off. No person on our side is included unless your agreement names one, and at launch the work is largely automated.
Stays with a person
- You or your named person accept every report and every change to a rule, mark or setting
- You apply anything the report suggests in your own systems
Access we would need
- Redacted, synthetic or public samples you send each month by an agreed secure route
- Your labels for the monthly sample
Questions
Do you run my step?
No. You send the golden set results and a monthly sample. We score them.
What if the score drops?
The report says so with the rows that changed. A fix is a separate agreed job.
Do I need the one-off golden-set job first?
Not if you already have a labelled golden set and thresholds.
Also part of
Send an enquiry
Send us
- Which step, which tool it runs in and how often it runs
- Whether you have a golden set already
- No real records, personal data or credentials in the first enquiry
Later, once you agree
- The golden set and thresholds file
- 20 fresh labelled rows each month by an agreed secure route
- The golden-set outputs for the month, from your harness run
- A named person who reads the report
- A written cap on model usage
You keep your step, your key and your data. We work on the golden set and the redacted sample you send each month, return a report, and delete our copies at the end of each month. We take no logins and no keys.
Email fallback: open your mail app
If website submission is unavailable, review and send the fallback email yourself. An email fallback is not a website receipt. Or write to hello@syntheticindustry.ai with “monthly-ai-job-drift-check-on-golden-set” as the subject.