Four kinds of case
This is our own method, not a standard. Write questions from your own pages and your old tickets. Include ones your pages answer, ones they do not, ones that must go to a person (refunds, complaints, legal, anything about one customer's account) and ones that try to make the bot misbehave.
Have the person who owns the policy approve the set and write the pass marks before the first run. After the first run you may add cases, never remove any.
- In scope: your pages answer it, and the bot should cite the page.
- Out of scope: no page answers it, and the bot should say so and offer a person.
- Handover: the bot should hand the conversation to a named person.
- Hostile: the bot should not state a price, refund or "legally binding" statement.
Run each case more than once
tau-bench found that policy-following agents often fail to repeat a task: an early model scored about 25% when the same retail task had to pass eight runs in a row. Run each case three times and call a case that passes only some runs flaky.
Read a pass carefully
Cases you write from your pages are a smoke test: a pass describes those cases, not a measured live pass rate. Say the sample size beside every pass mark and keep part of the set from the builder until the final run. List errors that fail the whole test, such as an invented price or refund.
A pass means the set behaved, not that the bot is accurate.
Why the fences matter
A tribunal held Air Canada responsible for what its chatbot told a customer and rejected the argument that the bot was a separate entity. A disclaimer is not a control; a refusal test and a named owner for the content are.
Sources and limits
- tau-bench: benchmarking tool-agent-user interaction (arXiv 2406.12045) Checked 2026-10-12.
- Policy-following agents often fail to repeat a task: an early GPT-4o scored about 61% on single runs in retail and about 25% when the same task had to pass eight runs in a row.
- Intercom Community: how do you train Fin AI to respond (13 Dec 2024) Checked 2026-10-12.
- A community member wrote that they had been testing Fin and were not yet satisfied enough with the responses to turn it on for all users.
- American Bar Association: B.C. tribunal confirms companies remain liable for information provided by AI chatbot Checked 2026-10-12.
- In Moffatt v Air Canada (2024 BCCRT 149) the tribunal rejected the argument that the airline's chatbot was a separate entity and held the airline responsible for the wrong fare rule it gave.