Which support tags to keep: count, merge, define and check that two people agree

What Apollo found

Apollo's engineers wrote that about 25,000 conversations arrive each month, about 8,000 were tagged by hand and there were over 248 tags, many unused or misapplied. Their model tagging scored macro-averaged F1 0.81 for category, 0.70 for feature and 0.50 for function against a set of many hundreds of labelled conversations, labelled by 10 to 15 customer success managers. Gains came from putting the taxonomy in a file, adding examples and summarising long threads, not from any single model improvement.

A routine you can do by hand

Export tag counts, sort by use and look at the long tail. Merge near-duplicates, give each kept tag a one-line definition and two real examples, and keep a map from each retired tag to the tag that replaces it.

• Count use of every tag over the last 90 days.
• Merge tags that mean the same thing.
• Write a definition and two examples for each tag you keep.
• Have two people tag the same 50 tickets and compare.

Check agreement before you test a tool

If two people cannot agree on the tag for the same tickets, a tool's score against either of them means little. Report raw agreement and which tags get confused. Zendesk's intelligent triage, a paid Copilot add-on, classifies topic, sentiment and language, so try it on the same 50 tickets if you have it.