Case study · Sales and marketing
A tool sorts the firm's incoming messages. Every month we check it still sorts them the way the owner would.
A tool sorts incoming messages and drafts replies. Each month we test that it still agrees with the owner, and that nothing has been sent.
- Who
- A professional services firm with eight staff, using a tool that sorts incoming messages and drafts replies.
- The problem
- The tool was tested once, on the day it was set up. The firm's services, prices and wording have changed since, and nobody had re-checked it.
- What we did
- Each month the firm's person labels 30 messages, and we compare the tool with their labels, re-run the traps and trace 20 drafts to the firm's answer sheet.
- The result
- A monthly report shows whether the tool still agrees with the firm, whether any trap got through, and which answers on the sheet are out of date.
- Time
- Ongoing. A monthly service.
What was going on.
The firm had a tool that read each incoming message from the website form and the shared inbox, labelled it as a new enquiry, an existing client, a supplier or something to ignore, and drafted a reply from the firm's answer sheet for a person to approve. When it was set up, it was tested and it worked.
Months later the firm had changed its prices, added a service and dropped another. The answer sheet had not kept up, and an update to the model underneath could change the tool's behaviour without anyone noticing. The test that passed on the day was never run again.
What we found.
A sorting tool drifts quietly as the business changes around it. The only defence is to run the same kind of test again, every month, with a person who knows the business marking the right answers.
Each month the firm's person labels 30 randomly chosen messages. We compare the tool's labels with theirs, re-run a set of trap messages built to trick it, trace 20 drafts line by line back to the answer sheet, and check that the log of sent messages is still empty, because the tool is only meant to draft.
The technical detail
The check uses 30 randomly chosen messages labelled by the firm's person, and at least 26 must match. That is a tripwire for drift and says nothing about how the tool will do on messages we have not seen. The trap messages and the messages that carry instructions must all be marked as needing a person. Every sentence in 20 sampled drafts must trace to a line in the answer sheet, and answer-sheet entries past their review date are listed.
What we did.
How we checked it.
These had to pass before we asked for sign-off.
- On 30 randomly chosen messages labelled by the firm's person, at least 26 get the same label, and any serious mistake is reported on its own.
- Every trap message is marked as needing a person, and the log of sent messages is empty.
- Every sentence in 20 sampled drafts traces to a line in the answer sheet.
What they have now.
Each month the owner reads a short report: how often the tool agreed with the firm, which traps it caught, which answer-sheet entries are past their review date, and any change we propose to the labels or the routing. The owner decides what to change.
In one month the report showed the tool treating a question about a discontinued service as a new enquiry, because the answer sheet still listed it. The firm updated the sheet.
What we did not do.
- We send no message to anyone. The tool drafts, and the firm's people approve and send.
- We do not edit the answer sheet. The firm owns it.
- The monthly agreement check says nothing about messages we have not seen.
- No cold or promotional messages, no regulated advice and no real-time cover.
The service behind this.
A sorting tool that passed its test once will drift. A small test every month tells you when it has.
Monthly service
Monthly message sorter test
Each month 30 of your messages are labelled by you and compared with the sorter, the traps are re-run and 20 drafts are traced to your answer sheet. The send log must stay empty.
$195/month. Billing is monthly under the written terms we agree with you. Nothing is charged until you have agreed them.
More case studies.
One-off job
Checkout stopped taking orders after an update
Customers could fill a basket but not pay. We found the cause on a copy of the shop, fixed it and proved it with test orders.
Online store · Two days
One-off job
Genuine payment messages were being rejected
Paid customers still showed as unpaid. The company's code was checking the wrong thing. We fixed it without switching the check off.
Software team · Three working days
Project
Hundreds of products flagged in Google's shopping feed
Several problems at once, and every earlier fix had only revealed the next one. We grouped them by cause and worked through them in order.
Online store · Three weeks, mostly waiting for Google to reprocess
Get a price for something like this
You are asking about: Monthly message sorter test.
Sending this is not an order. We check we can help, then agree what is included, safe access and the price in writing before any work. Prefer email? hello@syntheticindustry.ai.