Job support-bot-test-pack-run-before-you-switch-it-on · revised 12 October 2026
Run a written test of your support bot, with every fail shown, before you switch it on
A written test set run three times against your existing bot in a sandbox: every pass and fail shown with its transcript, the fix named for each fail, and one re-run.
You might be seeing
- Answers sound fluent but are vague, off-topic or wrong for the question
- The same question gets a different answer on another day
- Nobody has a written list of what the bot must refuse or hand over
No passwords, keys, card details or admin invites needed to start.
What usually happened
A support bot can sound sure while being wrong, and the same question can get a different answer the next day. Switching it on without a written test means customers find the failures first.
Who it’s for: Support lead about to turn on Fin, Lyro, a help-desk AI agent or a custom bot for all users.
Usually starts when: The bot gives answers that are technically right but not what the customer needs, and you do not yet trust it enough to turn it on for everyone.
The result: You receive a written test set run three times against your bot in a sandbox or staging chat, with every pass and fail shown beside its transcript, the page or setting to change for each fail, and one re-run after you have made the changes.
Check whether this job fits
Answer these without sending chats, tickets, files or logins. Nothing is submitted unless you choose to contact us.
What you get
- The test set and the three-run log
- The pass and fail list with transcripts
- The failure list by type
- The fix list naming the page or setting for each fail
- The re-run result
- A page on what the test does not show
Included
- One bot on one platform, tested in a sandbox, staging chat or test chats run from our written cases
- A test set of 100 cases you approve: 60 in scope, 15 out of scope, 15 that must be handed to a person and 10 hostile
- Each case run three times, with flaky cases reported as flaky
- A failure list by type: wrong, unsupported, answered out of scope, missed handover, automated label missing
- One re-run after your changes, and a page of what the test does not show
Not included
- Fixing your bot or rewriting your pages
- Switching the bot on or off for you
- Tests against live customer chats
- Security testing beyond the 10 hostile cases: see the injection and data test
- Any claim about resolution, deflection or customer satisfaction
- Advice on whether your bot meets disclosure law
How we know it’s done
Agreed with you before work starts. Each check produces evidence you keep.
You can re-ask 10 random cases yourself (seed recorded) and get the same verdict in at least 2 of 3 runs for the cases we list as stable; cases we list as flaky may also vary for you.
Evidence: The seed, the 10 cases and your own transcripts beside ours.
After your fixes, the final run meets the marks written before testing: at least 54 of 60 in-scope cases answer correctly, all 15 out-of-scope cases offer a person, all 15 handover cases reach your queue, and none of the 10 hostile cases produces a price, refund, discount or "legally binding" statement. A case counts as a pass only if all three runs pass and no severe error listed in the next check occurs. These authored cases are a smoke test, not a measured live pass rate.
Evidence: The re-run log against the written pass marks, with the 30 held-back cases shown separately.
Every fail has the page or setting to change named beside it. Any severe error in any run (an invented price, refund or policy, another customer's data, a claim to be a person, no route to a person) fails the whole test and is listed first.
Evidence: The fix list and the severe-error list.
The automated label and the way to reach a person are visible on the widget in the sandbox or staging chat, checked on a test chat.
Evidence: Screenshots of the test chat in each state.
Sign-off. You re-ask the 10 cases, read the fix list and the re-run, and sign off in writing. Payment follows sign-off.
If it fails. If the re-run does not meet the pass marks after your changes, we hand over the log and the open fails and you agree whether to retest or stop. A fix we cannot name is not claimed as done.
When it fits, and when we stop
It fits when
- You can give us a sandbox, staging chat or test workspace, or run our cases yourself and send the transcripts
- Your help pages or policies are current and have one named owner
- A named person on your side writes the pass marks and marks a sample
We stop and tell you if
- Only production access is available
- The bot has tools that write to customer records or send messages: run the injection and data test first
- The bot gives regulated advice
- Your pages contradict each other and nobody will settle which is right
What could go wrong
The test reads and chats in a sandbox only, so there is nothing to undo. Your bot, pages and settings are exactly as they were.
Scroll the table sideways to read it all.
| Risk | How we handle it |
|---|---|
| A pass is read as proof the bot is accurate | Every report says a pass means this set behaved, not that the bot is accurate, and lists what the set does not cover. |
| The same case passes once and fails the next time | Each case runs three times. A case that does not pass all three is reported as flaky and cannot count as a pass. |
| We tune the bot to our own test | Thirty cases are held back by your named person until the final run, and cases may be added after the first run but never removed. |
| Real customer transcripts reach us | The first enquiry takes no chats. Redacted history is used only after written agreement and a secure route. |
A separate automated reviewer checks the case list, the verdicts and the fix list before hand-over, and your named person on your side marks the sample and decides what goes live. No human supervisor is included unless your agreement names one. At launch the work is largely automated, and we say so.
How we deliver
We arrange the work and independent review, then show you the result against the agreed checks. You keep authority over your systems.
- Agree the bot, the pages, the pass marks and the severe-error list in writing
- You approve the 100 cases; your named person holds 30 back
- Run every case three times in the sandbox and record each transcript
- Sort every fail by type and name the page or setting to change
- Independent automated review of the verdicts; your person re-checks 10 cases
- Hand over the pack; you make changes and we re-run once
This is a one-off job, not emergency cover or a subscription. We confirm eligibility, the total price, a start window and a delivery date before you accept. Work starts only after agreed inputs, secure access and any needed permissions are in place. Model usage for our build and tests is included in the price; charges on your own platform account, including per-conversation fees after go-live, are yours. An enquiry creates no charge or booking.
Need to keep it working?
Discuss a monthly sample and re-test of the bot after you switch it on.
Ongoing work is separately scoped and quoted: no monitoring, response-time guarantee or automatic subscription is included in this job.
Explore an ongoing managed lane, or mention the responsibility you need in your enquiry.
What you can check
This is a new service. We have not delivered this job for a client yet.
Other ways to get this done
- Many help desks let you preview the AI agent against a few questions, so check yours. Use that first, and write down the questions that went badly.
- Ask five colleagues to try to break the bot with questions it should refuse, and keep the transcripts.
Questions
Does a pass mean the bot is accurate?
No. It means the bot behaved on this set of cases. We give the sample size and what the set does not cover.
Do you fix the bot?
No. We name the page or setting behind each fail. You make the change and we re-run once.
Why run each case three times?
The same question can get different answers on different runs, so a single pass does not show the bot is steady.
Send an enquiry
Send us
- Which bot and platform you use, and the kind of help centre behind it
- The pass marks you would want, in your own words
- Whether you can make a sandbox, a staging chat or test chats
- No customer data, logins or keys in the first enquiry
Later, once you agree
- Access to a sandbox or staging chat that you create and can revoke, or transcripts of test chats you run
- The page list or policy documents the bot should answer from
- Your approved case list, with 30 of the 100 held back by your named person
- A secure handoff agreed before any redacted ticket history is shared
You keep the bot, the pages and every login. We test in a sandbox or on test chats, change nothing live and take no customer records unless a redacted export is agreed in writing. Fixes are yours to make.
Email fallback: open your mail app
If website submission is unavailable, review and send the fallback email yourself. An email fallback is not a website receipt. Or write to hello@syntheticindustry.ai with “support-bot-test-pack-run-before-you-switch-it-on” as the subject.