Job past-ticket-examples-versus-help-pages-which-gives-better-reply-drafts · revised 12 October 2026
Find out whether your past tickets beat your help pages as the source for reply drafts
A blind comparison on 60 held-out tickets: reply drafts built from your help pages alone against drafts that also see examples of resolved tickets. You choose which source to use.
You might be seeing
- A vendor asks for your whole ticket history
- You cannot tell if past tickets would help or just add risk
- Past replies contain outdated policies and other customers' details
No passwords, keys, card details or admin invites needed to start.
What usually happened
Past tickets can teach a drafting assistant your tone, but they also hold old policies and other customers' details that can leak into new drafts. Few buyers have seen a head-to-head test on their own tickets.
Who it’s for: Support lead told by a vendor that the assistant must be "trained on your tickets", who wants to see whether that helps.
Usually starts when: A vendor says your past tickets will make drafts better, and you worry about sending customer conversations into a model for training.
The result: You receive a blind comparison on 60 held-out redacted tickets of reply drafts built from your help pages alone against drafts that also see ten example resolved tickets per type. Your two best agents rate them, and you get the counts of wins, ties, losses, invented claims and copied details. We train and fine-tune nothing on your tickets; the redacted tickets are shown to a hosted language model for the test, and the provider, country and retention are named in writing before any export is shared.
Check whether this job fits
Answer these without sending chats, tickets, files or logins. Nothing is submitted unless you choose to contact us.
What you get
- The comparison results with all counts
- The blind rating sheet
- The list of any copied details or invented claims
- A recommendation page
Included
- One help desk, ten ticket types and your help pages
- Two drafting methods compared on 60 held-out redacted tickets
- Blind ratings by your two best agents
- Counts of invented claims and of details copied from an example ticket
- A plain recommendation of which source to use, or neither
Not included
- Training or fine-tuning any model on your tickets
- Building the drafting assistant: see the reply drafts job
- Sending any reply
- Tickets with special-category data
- Any claim that drafts save time
How we know it’s done
Agreed with you before work starts. Each check produces evidence you keep.
Your two agents rate all 60 pairs blind and in random order, and the report shows wins, ties and losses for each method with the raw counts. A pair is a win only when both agents prefer the same draft; otherwise it is a tie.
Evidence: The rating sheet with the method key kept sealed until after rating.
Each method's drafts are scanned for invented policies, prices and dates, and for names or details copied from an example ticket. Your agents check every flagged invented claim. Any copied identifying detail is reported first, and a method with one cannot be recommended.
Evidence: The scan results per method.
The recommendation names a source only if it wins at least two thirds of the pairs that are not ties, at least 30 pairs differ (20 wins of 30 gives a sign-test p of about 0.05) and it has no more invented claims; otherwise it says neither is clearly better.
Evidence: The recommendation page with the counts it rests on.
Sign-off. Your agents read the results and the recommendation and you sign off in writing. Payment follows sign-off.
If it fails. If the ratings cannot be completed or the tickets cannot be redacted safely, we say so and agree whether to stop or re-quote. A tie is a valid result.
When it fits, and when we stop
It fits when
- Current help pages exist with one named owner
- About 300 resolved tickets can be redacted by you and shared
- Two agents can rate drafts blind
We stop and tell you if
- Tickets hold special-category data or cannot be redacted
- You want a model fine-tuned on your tickets
- Nobody can rate the drafts
What could go wrong
The test reads redacted exports and writes a report, so nothing in your systems changes. We delete our copies at hand-over and train or fine-tune nothing on them.
Scroll the table sideways to read it all.
| Risk | How we handle it |
|---|---|
| A draft copies a detail from an example ticket into a new one | Every draft is scanned for names, numbers and phrases from the examples, and one copied identifying detail counts as a severe error. |
| The examples contain an outdated policy | Invented or outdated claims are counted per method and reported. |
| The comparison is too small to decide | The report gives the counts, says how many pairs differed and states when the result is a tie. |
A separate automated reviewer checks the draft pairs, the counts and the copied-detail scan before hand-over, and your two agents on your side mark the sample and decide what goes live. No human supervisor is included unless your agreement names one. At launch the work is largely automated, and we say so.
How we deliver
We arrange the work and independent review, then show you the result against the agreed checks. You keep authority over your systems.
- Agree the ticket types, the held-out 60 and the severe-error list in writing
- Build drafts from your help pages alone for the 60 tickets
- Build drafts from the same pages plus ten example tickets per type
- Two of your agents rate all pairs blind and in random order
- Count invented claims and details copied from example tickets
- Independent automated review; hand over the results and recommendation
This is a one-off job, not emergency cover or a subscription. We confirm eligibility, the total price, a start window and a delivery date before you accept. Work starts only after agreed inputs, secure access and any needed permissions are in place. Model usage for our build and tests is included in the price; charges on your own platform account, including per-conversation fees after go-live, are yours. An enquiry creates no charge or booking.
Need to keep it working?
Discuss building the reply drafts with the better source.
Ongoing work is separately scoped and quoted: no monitoring, response-time guarantee or automatic subscription is included in this job.
Mention the responsibility you need in your enquiry.
What you can check
This is a new service. We have not delivered this job for a client yet.
Other ways to get this done
- Ask the vendor for a test on 60 of your tickets with its examples switched off and on, and rate the results blind yourself.
Questions
Do you train a model on my tickets?
No. We train and fine-tune nothing on your tickets. The redacted tickets are shown to a hosted language model for the test, with the provider, country and retention named in writing before any export is shared, and we delete our copies afterwards.
What if neither method is clearly better?
Then the report says neither is clearly better and gives the counts; you can keep your help pages alone, the simpler source.
Does this build the assistant?
No. The reply drafts job does.
Send an enquiry
Send us
- Your help desk and the ten ticket types that matter most
- Where your help pages live
- What the vendor proposed
- No customer data, logins or keys in the first enquiry
Later, once you agree
- About 300 resolved tickets, redacted by you, through a secure route agreed in writing; 60 chosen by your lead and held back
- An export of the help pages
- Two agents to rate the drafts blind
You keep the tickets, the pages and every login. We work on redacted exports and keep no copies after hand-over. We train and fine-tune nothing on your tickets; the redacted tickets are shown to a hosted language model for the test, and the provider, country and retention are named in writing before any export is shared.
Email fallback: open your mail app
If website submission is unavailable, review and send the fallback email yourself. An email fallback is not a website receipt. Or write to hello@syntheticindustry.ai with “past-ticket-examples-versus-help-pages-which-gives-better-reply-drafts” as the subject.