The question: a green test suite tells you the tests passed. It does not tell you
the tests would have noticed if the code were wrong. This page finds out, by breaking
game.html on purpose - one seeded defect at a time - and reporting how many of
them the game's own suite catches.
Every mutation is an exact string replacement listed in
tools/mutations.json. For each one, this page rebuilds the whole file in
memory, runs all of its tests in an offscreen iframe, and records whether any test failed.
A defect that no test notices is a hole, and it is named below.
The iframe runs same-origin, not sandboxed - the game's tests need real
localStorage to exercise saving. So this page takes every aiquest*
key out of storage before each mutant and puts it back afterwards. Without that, one mutant's
boot could change the next one's result, and a save-subsystem defect could eat a real save.
This page needs to be served over http so it can fetch the game file.
From the repository root: python -m http.server 8000, then open
http://localhost:8000/tools/mutation-lab.html. Opening it straight off disk
will fail at the first fetch.
| id | outcome | what it breaks | which test noticed |
|---|
KILLED means at least one test failed when the defect was present, or the defect crashed the page before any test could run - a crash is the most visible failure there is, so it counts. SURVIVED means every test passed with a real defect in the code. Those are the interesting rows: each is something this project believes and does not check. NOT APPLIED means the search string did not match exactly once, which is a fact about the catalogue rather than the suite, so those rows leave the denominator.
What a kill does not prove. Many of this suite's tests are greps over the file's own source text, so some kills mean "a check noticed the source changed" rather than "a behavioural assertion failed". A textual kill gives no assurance against the same defect spelled a different way.
What a survivor does not prove either. No attempt is made here to exclude equivalent mutants - defects that change the source without changing observable behaviour - or defects sitting on states a real playthrough cannot reach. Both inflate the survivor count, so treat the kill rate as a floor.
The kill rate is not a score to maximise by deleting hard mutations. The catalogue is committed, so it can only grow, and every survivor stays in the file after it is fixed - a fixed hole that quietly left the catalogue would make the next number a lie. And a rate measured against a catalogue the suite was patched against is training-set performance, not a detection rate; see the ledger's correction on exactly that mistake.