The difference in one example
Coverage counts lines the tests ran. A mutation tool changes the code in small ways, for example flips a comparison, and counts how many changes make a test fail.
A test that runs a line but asserts nothing about it adds coverage and catches no mutation.
Why this matters for AI-written tests
In one case, research found language-model tests reaching near-total coverage with a very low mutation score. Feeding mutation results back raised the score.
Meta reported that its engineers accepted 73% of the tests an AI tool generated, but only 36% were judged relevant to the privacy concern they targeted.
What to check yourself
Pick three behaviour changes in a module, for example change a limit, drop a check, swap an operator. Run the tests. Each change should make at least one test fail.
Then open ten tests at random and ask whether each asserts something about behaviour or only that the code ran.
- Three changes you pick
- Ten tests read by a person
Limits
Some mutations change nothing observable and cannot be caught; a good report lists them. Stryker supports JavaScript, TypeScript, C# and Scala; other languages need an equivalent tool. A high score pins current behaviour, which may include a bug.
Sources and limits
- Meta Engineering: LLMs are the key to mutation testing Checked 2026-10-12.
- Meta's privacy engineers accepted 73% of the tests an AI tool generated, with 36% judged privacy relevant.
- Stryker mutator Checked 2026-10-12.
- Stryker supports JavaScript, TypeScript, C# and Scala.
- arXiv: language-model test generation and mutation scores Checked 2026-10-12.
- Research found, in one case, language-model tests reaching 100% line and branch coverage with a 4% mutation score, and that feeding mutation results back raised the score.