A long tag list is the first problem
Apollo's engineers describe about 25,000 conversations a month with about 8,000 tagged by hand and over 248 tags, many unused or misapplied. Start by counting how often each tag is used and merging the near-duplicates. A model cannot be better than the list it is given.
Check that two people agree before you test a tool
If two of your staff cannot agree on the tag for the same 50 tickets, no pass mark for a tool means anything. Fix the definitions first. Report raw agreement and which categories get confused.
Expect deeper levels to be weaker
Apollo's tagging scored F1 0.81 for category, 0.70 for feature and 0.50 for function. Accept a deeper level only if it meets its own mark on tickets your staff labelled. Zendesk's intelligent triage, a paid Copilot add-on, classifies topic, sentiment and language; if you have it, try it on 50 tickets first.
Which paid job fits
The tag list job cuts the list and writes definitions. The suggestion job tests category, priority and queue suggestions on 200 held-out tickets and never routes live until you turn it on. The monthly check re-tests on fresh tickets.
None scores your agents.
Sources and limits
- Apollo engineering blog: AI tagging from feedback to actions (26 Aug 2025) Checked 2026-10-12.
- About 25,000 Intercom conversations arrive each month and about 8,000 were tagged by hand, with over 248 tags, many unused or misapplied.
- Model tagging scored macro-averaged F1 0.81 for category, 0.70 for feature and 0.50 for function against a set of many hundreds of labelled conversations, labelled by 10 to 15 customer success managers, and gains came from taxonomy-as-YAML, examples and summarising long threads rather than any single model innovation.
- Zendesk: about intelligent triage Checked 2026-10-12.
- Zendesk's intelligent triage, a paid Copilot add-on, classifies topic, sentiment, language and custom entities.