Shape is not correctness
Structured output settings make the answer fit a schema. They do not tell you the values are right. A step can pass every shape check and still be wrong.
Label a golden set and grade in code
Pick 100 inputs with known right answers, labelled by people who agree with each other. Grade by exact match and field checks, never with another model. Run your unchanged step five times on your own key and expect the same verdict, or a floor set below the spread.
- Two people label 50 rows and agree on at least 43.
- Five planted regressions must fail, and you choose two of them.
- Three harmless changes must pass.
Expect variance
A 2023 study found that repeating an identical input or rewording a prompt can change labels. That is why exact-match grading and repeat runs matter, and why the test is built around cases that must never flip.
Which job fits
The golden-set job builds the set and the harness once. The monthly drift check scores the outputs of the set plus 20 fresh rows you label, to catch changes you did not make; 20 rows catch only large drifts. You keep your own provider key throughout.
Sources and limits
- OpenAI: Structured Outputs Checked 2026-10-12.
- Says the feature ensures responses adhere to a supplied JSON Schema, and that a refusal or a token limit can leave a response that does not match; it describes structure, not whether values are correct.
- arXiv 2304.11085: Testing the Reliability of ChatGPT for Text Annotation and Classification Checked 2026-10-12.
- Reports that minor wording changes in prompts or repeating an identical input can lead to varying outputs, and does not recommend unsupervised use for text annotation and classification.