How to use this order
Work through the steps and stop at the first one you cannot pass. This is an authored sequence, not a vendor procedure, and every step can be done by your own team. Use invented customers for anything that could reach a real person.
Steps one to three: gaps, facts, new articles
Step 1, find the gaps: a community member wrote that Fin's answers were technically correct but often not relevant, and a reply said such answers often point to a gap in the help centre. Gartner reported that the most common reason for self-service failure was that 43% of customers could not find content relevant to their issue. Step 2, check each article against your current prices and policies.
Step 3, write the missing articles with every claim traced to a source.
- Step 1: ranked list of questions no page answers.
- Step 2: each article marked current, outdated or in conflict.
- Step 3: new articles with a claim-to-source table.
Steps four to six: test, hand over, try to break it
Step 4, run a written test of the bot three times and fix what fails. Step 5, check every handover trigger reaches a named person with the whole conversation. Step 6, try hostile messages in a sandbox: OWASP lists prompt injection and excessive agency as the first risks, and recommends least privilege and human approval for sensitive actions.
In Moffatt v Air Canada the airline was held responsible for what its chatbot said, so the fences matter.
- Step 4: test set, three runs, a fix named for every fail.
- Step 5: handover cases land in the right queue.
- Step 6: hostile inputs fail to move the limits.
Step seven: go live yourself, then keep checking
You switch the bot on, label it as automated and keep the off switch. A monthly sample of real conversations and a re-run of the test catch drift when pages, prices or the vendor's model change.
What to buy, if anything
No bot yet? The chatbot job sets one up and includes its own test of 100 questions plus 15 hostile ones.
Each step has a one-off job, and the monthly bot check and help centre upkeep keep it true afterwards. A lane bundles the two monthly services with a small allowance of the one-off jobs. Nothing here promises fewer tickets, and a person on your side makes every live change.
Sources and limits
- Gartner via CX Today: only 14% of customer service issues are fully resolved in self-service Checked 2026-10-12.
- Gartner found only 14% of customer service issues were fully resolved in self-service, 36% even for very simple issues, and the most common reason for self-service failure was that 43% of customers could not find content relevant to their issue.
- Intercom Community: improving Fin AI responses to vague queries (5 Dec 2024) Checked 2026-10-12.
- A community member wrote that Fin responds with technically correct information that is often not relevant to what the customer needs, and a reply said such answers often point to a gap in the help centre.
- American Bar Association: B.C. tribunal confirms companies remain liable for information provided by AI chatbot Checked 2026-10-12.
- In Moffatt v Air Canada (2024 BCCRT 149) the tribunal rejected the argument that the airline's chatbot was a separate entity and held the airline responsible for the wrong fare rule it gave.
- OWASP Top 10 for LLM Applications 2025 Checked 2026-10-12.
- LLM01 is prompt injection and LLM06 is excessive agency, and the listed mitigations include least privilege and human approval for sensitive actions.