Two numbers for the same test
A field is one value on an invoice, such as the date. An invoice is right in full only if every field is right. In one small public test, two prompt strategies tied at 91.7% pooled field accuracy, yet 60% and 75% of invoices were fully right.
- Zero-shot: 24 of 40 invoices fully right, 91.7% of fields.
- Few-shot: 30 of 40 invoices fully right, 91.7% of fields.
- Rules: 35 of 40 invoices fully right, 99.0% of fields.
How much to trust that test
It is a single author's test of 60 synthetic invoices, with 40 used for scoring and one model. The author says the intervals are wide and that part of the best result may come from the generator's own conventions. Treat it as a reason to ask for the whole-invoice number, not as a rate for your invoices.
What an arithmetic check catches
Checking that lines plus tax equal the total flagged half of the wrong zero-shot extractions and all of the wrong few-shot and rules ones, with no false alarms. It cannot see a wrong number that is consistent: a price change worked consistently through the totals was missed in 49 of 49 trials. So the check is necessary and not enough.
What to ask for
Ask for a test on invoices you labelled and the tool has not seen. Ask for the whole-invoice rate, the per-field rates, the exceptions list and what happens when a field is missing. Remember that most invoice exceptions, according to one benchmark summary, come from purchase-order mismatches and vendor master data, not from reading.
- Whole-invoice rate on held-back invoices.
- Missing fields come back empty.
- Files in = invoices out + exceptions.
Sources and limits
- GitHub: LLM-invoice-extraction Checked 2026-10-12.
- The test used 60 synthetic invoices (5 layouts times 12), 20 for development and 40 for testing, with one model, scored once per strategy.
- All fields correct: zero-shot 24 of 40 (60.0%), few-shot 30 of 40 (75.0%), rules 35 of 40 (87.5%); pooled field accuracy 91.7%, 91.7% and 99.0%.
- An arithmetic validator flagged 8 of 16 wrong zero-shot extractions and every wrong few-shot (10 of 10) and rules (5 of 5) extraction, with no false alarms; a price change recomputed consistently through the totals went undetected in 49 of 49 trials.
- The author notes the data is synthetic, only one model was used, n=40 gives wide confidence intervals, and the rules prompt reflects the generator's own conventions.
- Parseur: AI invoice processing benchmarks Checked 2026-10-12.
- States that most invoice exceptions are not extraction failures and attributes them mainly to purchase-order mismatches, missing receipts, tax and freight discrepancies, duplicate checks and vendor master data.