A field rate is not a document rate
If each of 15 fields is right 97% of the time and errors are independent, 0.97 to the power of 15 is about 0.63. So about 37% of documents hold at least one error, which a vendor blog rounds to 36%. A tool described as "97% accurate" can therefore still be wrong somewhere on more than a third of invoices.
Count documents, and say what a pass shows
A pass mark means little without the sample size. At 50 items, 45 passes shows only that the true rate is at least 80% with 95% confidence. At 100 items, 96 passes shows at least 91%, 95 passes shows at least 89%, and 90 passes shows at least 83%.
Computed by Synthetic Industry: exact one-sided binomial lower bounds at 95% confidence, rounded down.
- State the number of items, the pass mark and the held-back share.
- List the severe errors that fail the whole test.
- Report documents passed, not fields passed.
What the errors look like
Published work on language-model extraction describes confusing similar entities, attaching a value to the wrong field and missing information. These errors can look plausible on the page, which is why a doubtful field should be held for a person rather than guessed.
How our jobs state it
Each reading job gives the number of documents you label, the pass mark, the part held back until the test run and the errors that fail the whole test. The result describes documents like the sample. It is not a promise about future documents.
Sources and limits
- Invoice data extraction: OCR accuracy developer guide Checked 2026-10-12.
- A vendor blog says header fields read at 99% or better and line items at 95 to 97%, and that at 97% per field across 15 fields about 36% of documents contain at least one error.
- arXiv 2603.18014: errors in language-model extraction Checked 2026-10-12.
- The paper describes errors such as confusing similar entities, attaching values to the wrong field and missing information, and reports document-level as well as field-level accuracy.