Job conversation-quality-review-against-your-rubric · revised 12 October 2026
Score a sample of past support conversations against your own written rubric
A sample of past conversations scored on your own rubric of up to ten criteria, with the worst examples, coaching notes and a stated agreement rate with your QA lead.
You might be seeing
- Only a tiny share of conversations is ever reviewed
- Reviewers score the same conversation differently
- Coaching notes are anecdotes, not examples
No passwords, keys, card details or admin invites needed to start.
What usually happened
Manual quality review covers a tiny sample, so patterns show up late. Automated scores that nobody has compared with a reviewer can mislead, and scores used on individuals raise legal and fairness risks.
Who it’s for: Support lead who reviews only 1 to 2% of conversations by hand and wants a wider, checkable read.
Usually starts when: Quality problems surface only when a customer complains, and the manual review sample is too small to show patterns.
The result: A sample of past conversations is scored on your written rubric, with the worst examples and coaching notes at team or topic level. A stated agreement rate with your QA lead is reported, and criteria where agreement is below your mark are dropped from the report.
Check whether this job fits
Answer these without sending chats, tickets, files or logins. Nothing is submitted unless you choose to contact us.
What you get
- The rubric as agreed
- Scores at team or topic level
- The worst examples and coaching notes
- The per-criterion agreement table
Included
- A rubric of at most ten criteria that you write or approve
- Up to 300 redacted written conversations, scored per criterion
- Results at team or topic level, with the worst examples
- A per-criterion agreement table against 20 conversations your QA lead scores blind
- Criteria below your agreement mark dropped from the report, not reported
Not included
- Scores for named individual agents
- Anything used for pay, ranking, discipline or dismissal
- Voice analysis or emotion inference about agents
- Customer satisfaction predictions
- Changes to your help desk
How we know it’s done
Agreed with you before work starts. Each check produces evidence you keep.
Your QA lead re-scores 20 conversations blind and agrees with our score on a criterion (the same pass or fail against the rubric line you wrote) on at least 17 of the 20 (17 of 20 shows a true agreement rate of at least 65% with 95% confidence, which is a weak bound); criteria below your mark are dropped from the report.
Evidence: The agreement table per criterion with raw counts.
Every score carries a quoted reason from the conversation, and the worst examples are listed with their reasons.
Evidence: The scored list with quoted reasons.
No report line names an individual agent, and the example conversations are redacted.
Evidence: The report checked for agent names.
Sign-off. Your QA lead reads the agreement table and the report and signs off in writing. Payment follows sign-off.
If it fails. If fewer than half the criteria reach your agreement mark, we hand over what survives and the dropped list and you decide whether to rewrite the rubric or stop.
When it fits, and when we stop
It fits when
- Written transcripts can be exported with personal details removed
- You have or will write a rubric of at most ten criteria
- A QA lead can score 20 conversations blind
We stop and tell you if
- You want scores for individual agents used for pay, ranking or dismissal
- The source is call audio or voice emotion
- Your QA lead says the written rubric cannot be applied to the 20 conversations as it stands: the rubric is rewritten first
What could go wrong
The review reads an export and writes a report, so nothing in your help desk changes. Our copies of the export are deleted at hand-over.
Scroll the table sideways to read it all.
| Risk | How we handle it |
|---|---|
| Scores are used to judge individual agents | The report is at team or topic level, agent names are masked, and the stop condition refuses pay, ranking or discipline uses. |
| A score disagrees with how a human would read the conversation | Agreement per criterion is shown and criteria below your mark are dropped, not reported. |
| Staff are monitored without being told | Telling staff about the review is your duty; our report includes a note saying so. |
A separate automated reviewer checks the scores, the quoted reasons and the agreement table before hand-over, and your QA lead on your side marks the sample and decides what goes live. No human supervisor is included unless your agreement names one. At launch the work is largely automated, and we say so.
How we deliver
We arrange the work and independent review, then show you the result against the agreed checks. You keep authority over your systems.
- Agree the rubric, the agreement marks and the team or topic groups
- Your QA lead scores 20 conversations blind and holds them back
- Score the sample on every criterion with a quoted reason
- Compare with your lead's 20 and compute agreement per criterion
- Independent automated review; drop criteria below your mark
- Hand over the report at team or topic level
This is a one-off job, not emergency cover or a subscription. We confirm eligibility, the total price, a start window and a delivery date before you accept. Work starts only after agreed inputs, secure access and any needed permissions are in place. Model usage for our build and tests is included in the price; charges on your own platform account, including per-conversation fees after go-live, are yours. An enquiry creates no charge or booking.
Need to keep it working?
Discuss a repeat review each quarter on a fresh sample.
Ongoing work is separately scoped and quoted: no monitoring, response-time guarantee or automatic subscription is included in this job.
Mention the responsibility you need in your enquiry.
What you can check
This is a new service. We have not delivered this job for a client yet.
Other ways to get this done
- Raise your manual review from 1% to 3% with a one-page rubric and a spreadsheet before buying anything.
Questions
Can I get a score for each agent?
No. Results are at team or topic level and agent names are masked.
Is this a legal review of monitoring staff?
No. You are responsible for telling staff and checking worker-monitoring rules in your country.
Do you listen to calls?
No. Written conversations only.
Send an enquiry
Send us
- Your help desk and the channels in scope
- The rubric you use today, or the qualities you care about
- How many conversations you handle in a month
- No customer data, logins or keys in the first enquiry
Later, once you agree
- A redacted export of up to 300 written conversations through a secure route agreed in writing
- The approved rubric, with the agreement mark for each criterion
- A QA lead to score 20 conversations blind
You keep the help desk and every login. We work from a redacted export and give you a report. Staff names are removed or masked before we see the export.
Email fallback: open your mail app
If website submission is unavailable, review and send the fallback email yourself. An email fallback is not a website receipt. Or write to hello@syntheticindustry.ai with “conversation-quality-review-against-your-rubric” as the subject.