Standing service support-bot-answers-sampled-and-retested-every-month · revised 12 October 2026
Standing service
Have a sample of your support bot's real conversations read and re-tested every month
Each month a fixed sample of your bot's real conversations is read, errors are counted by type, the test set is re-run after any change you report, and you get a fix list.
This starts a conversation by email. Nothing is charged, and nothing is monitored, until we have agreed scope, access and terms with you in writing.
The responsibility you hand over
A bot that passed a test at launch keeps changing under you: pages are edited, prices move and the vendor updates its model. Nobody reads the conversations, so a wrong answer repeats for weeks.
Who it’s for: Support lead whose bot was good at launch and drifts as help pages, prices and the vendor's model change.
Usually starts when: Customers are quoting the bot back to your team, or the bill for "resolved" chats looks higher than the chats you can see.
The result: Every month a random sample of your bot's real conversations is read, errors are counted by type, every handover trigger and 10 hostile inputs are re-run, the saved test set is re-run after any change you report, and you receive the fix list with the page or setting to change for each error.
What stays true, and what we do about it
No response-time guarantee is published for this new service. A date for each monthly report is agreed in writing before it starts.
Hours are agreed in writing before the service starts. It runs once a month after month end and is not staffed round the clock.
What must remain true
- Each month a random sample of 30 of your bot's conversations is read and every error is counted by type: wrong, unsupported, answered out of scope, missed handover, unsafe commitment
- The saved test set still behaves as the pass marks you wrote down after any change you report to pages, settings or the vendor's model
- Every handover trigger on your list still reaches the named person with the conversation
What we watch
- A read-only export or saved view of the bot's conversations for the month
- Your list of changes to pages, policies and bot settings
- The saved test set, handover list and the hostile inputs
When something happens
Scroll the table sideways to read it all.
| When | What we do |
|---|---|
| A month closes and the transcript export is available. | We draw the random sample, classify the errors, re-run the handover tests and send the report. |
| You tell us a page, policy or price changed. | We re-run the affected test cases and list any answer that now conflicts with the change. |
| The vendor changes the model or a bot setting. | We offer a full re-run of the test pack and report which cases changed. Linked job: Support bot test pack |
| The same wrong answer appears in two months running. | We flag it first in the report and name the page or rule behind it. |
We do on our own
- Read the sampled conversations and classify the errors
- Re-run the saved test set and handover tests in the sandbox
- Add failed cases to the test set, never remove one
- Write the report and the fix list
We ask you first
- Any change to the pass marks or the error types
- Any request for wider transcript access
- Changing the bot, the pages or any setting
- Contacting a customer
- Adding a channel such as WhatsApp or voice
We escalate to you when
- A conversation shows a refund, price or policy that is on none of your pages
- A legal threat, a data-access request or a safety concern appears in a conversation
- A leak of another customer's data or a successful hostile instruction is suspected
- The same wrong answer appears twice in a month
- The bot platform changes its AI or messaging terms
How you know it held. Each month you can re-read 10 of the sampled conversations and agree with the classification, and the report states the sample size and what it cannot show.
How we keep it true
This service is never finished. Each month's report shows whether the things it watches stayed true, and it continues until you end it.
Run a written test of your support bot, with every fail shown, before you switch it on Job Optional
A written test set run three times against your existing bot in a sandbox: every pass and fail shown with its transcript, the fix named for each fail, and one re-run.
Bought separately, each time it is needed; not part of the monthly fee
The one-off test pack you can buy on its own; used to rebuild the saved test set after a big change.
Make sure your bot hands customers to a named person with the whole conversation attached Job Optional
Each listed trigger sends the customer to a named person or queue with the full conversation and a tag, and the out-of-hours message states only hours you can keep.
Bought separately, each time it is needed; not part of the monthly fee
The one-off handover set-up for adding or changing triggers.
Test what a hostile message can make your support bot reveal or do, with a permission list Job Optional
A record of 40 hostile inputs run five times each against your bot in a sandbox, a list of what the bot can see and do, and the permission to remove for each failure.
Bought separately, each time it is needed; not part of the monthly fee
The one-off hostile-input test when you give the bot a new tool.
What is included, and what is not
- A monthly report with counts by error type
- The fix list with the page or setting for each error
- New test cases made from the failures
- The handover re-test result
- The result of the 10 hostile inputs, with any that revealed data or made a commitment listed first
Included
- One bot on one platform and its named handover queue
- A random sample of 30 conversations each month (seed recorded), read and classified by error type
- A re-run of the saved test set after any change you report and a re-test of every handover trigger
- Ten hostile inputs re-run each month against the sandbox
- A monthly report with the sample size, counts, fix list and what the report cannot show
Not included
- Any resolution rate, deflection rate, satisfaction score, cost or headcount figure
- Changing the bot, the pages or any setting: your named owner does that
- Contacting customers
- Round-the-clock cover or a response-time guarantee
- Voice or messaging channels we have not agreed
How we know it’s done
Agreed with you before work starts. Each check produces evidence you keep.
You can re-read 10 sampled conversations (seed recorded) and agree with the classification in the report on at least 9 of them.
Evidence: The seed, the 10 conversations and your own marks beside ours.
The report states the sample size, the count of each error type, the handover test result, the result of the 10 hostile inputs (any that revealed data or made a commitment listed first) and what it cannot show, and every error has a page or rule fix named.
Evidence: The monthly report.
The handover triggers are re-tested end to end each month and any trigger that fails is listed first.
Evidence: The handover re-test table.
No change, reply or message has been made by us in your bot or help desk.
Evidence: Your platform's audit trail for the month, which shows no writes by our access.
Sign-off. You read each report, re-read the sample and tell us whether the pass marks and error types still suit you. A month counts as delivered when you have the report and fix list.
If it fails. If you agree on fewer than 9 of the 10 conversations you re-read, we re-classify that month's whole sample and show every change. If we cannot sample a month because the export is missing or has changed, we say so and do not guess. If the service is not working for you, you can end it at the end of any month.
When it fits, and when we stop
It fits when
- Read-only access to the bot's conversations as a saved view or export, handled under written terms
- A kill switch that you hold
- A named person on your side who reads the report and changes pages or settings
- A saved test set, or you buy the test pack first
We stop and tell you if
- You refuse transcript access: the service falls back to synthetic-only checks
- Most errors come from causes you will not change
- Nobody on your side can change the pages or settings
What could go wrong
The service reads and reports, so there is nothing to undo. Ending it leaves your bot exactly as it is; you keep every report.
Scroll the table sideways to read it all.
| Risk | How we handle it |
|---|---|
| A sample of 30 misses a rare error | The report states the sample size and that a sample is a tripwire: it can show gross failure, not small error rates. |
| The report is read as a resolution or deflection figure | We never state those figures. The report counts errors by type and says what it cannot show. |
| Transcripts carry personal data | Access is read-only under written terms, redaction is done by you where possible, and the provider, its country and its retention are named in writing first. |
| Hostile text in a conversation steers the reader | Conversation text is treated as data, never as instruction, and nothing is sent to anyone from the sample. |
A separate automated reviewer checks each report, and a person on your side re-reads a sample. No human supervisor is included unless your agreement names one. At launch the work is largely automated, and we say so.
Stays with a person
- You or your named owner change every page, rule and setting
- You hold the kill switch for the bot
Access we would need
- A read-only transcript export or saved view that you generate
- A sandbox or test workspace for the re-runs
Questions
Do you tell me my resolution rate?
No. We count errors by type in a sample and report the sample size. We never state resolution, deflection or satisfaction figures.
Do you change my bot?
No. We name the page or setting behind each error. You or your named owner make the change.
What if I do not want to share transcripts?
The service falls back to synthetic-only checks, which are weaker. The report says so.
Also part of
Send an enquiry
Send us
- The bot platform and roughly how many conversations it has each month
- Who would read the monthly report
- Whether a test set already exists
- No customer data, logins or keys in the first enquiry
Later, once you agree
- A read-only transcript export or saved view that you set up and can revoke, with personal details removed where you can
- The saved test set and handover list, or the order for the test pack
- The written terms for handling real transcripts, agreed before any export
You keep the bot, the pages and every login. We read a transcript view or export that you generate, work in a sandbox for the re-runs and never change your live bot or reply to a customer. You hold the kill switch.
Email fallback: open your mail app
If website submission is unavailable, review and send the fallback email yourself. An email fallback is not a website receipt. Or write to hello@syntheticindustry.ai with “support-bot-answers-sampled-and-retested-every-month” as the subject.