Tell a renamed test from an unbound guard in the mutation ratchet - #854
Merged
Conversation
…althy one
`vitest run <file> -t <name>` exits 0 when the name matches nothing. It skips
every test in the file and reports success. The ratchet read only that exit code,
so a registration pointing at a test that no longer exists was indistinguishable
from a mutation nothing reacted to, and both came back `inert`.
Renaming a test is routine. A/B on the same broken registry, with the test name
of an inert-baselined guard changed to one that does not resolve:
old exit 0 64 bound, 1 inert, 1 unavailable, 13 uncovered no failures
new exit 1 64 bound, 0 inert, 1 unresolved, 1 unavailable REGISTRATION DEFECT
The old summary is character-for-character what a healthy run prints. Nothing
anywhere said the guard had stopped being checked.
The second gap has already cost a session. Yesterday four v6 guards were rejected
and two were reported `inert`; applying those mutations by hand failed a test both
times, a different test than the registry named. The ratchet's message says the
property has no test defending it, but what it observes is that the *named* test
did not fail — the same output for an uncovered guard and for a misfiled one, and
the misfiled one looks fine on inspection because the mutation is real and the
test is real.
So the reading now distinguishes four things where it used to see two:
bound the registered test failed
misfiled it passed, but the mutation failed some other test — named
unresolved the registered name matched nothing, so nothing was measured
inert the mutation ran and no test in the file failed
`misfiled` and `unresolved` are not coverage gaps, they are broken registrations,
so `bench/cdeb/guards/baseline.json` may not hold them and measuring one fails on
sight. A recorded gap says how far coverage reaches and can be carried with a
reason; a broken registration says the recorded coverage cannot be checked at all,
and ratcheting that in defeats the baseline.
The unfiltered second run costs one extra Vitest process, and only on the path
where the named test survived — two of sixty-six here. Bound mutations, the
common case, still cost one.
The reading moved to `scripts/guard-outcomes.mjs` because the ratchet spawns a
Vitest process per mutation and cannot be exercised from inside the suite it
runs. The decision table is pure, so `test/guard-ratchet-outcomes.test.ts` covers
it in two milliseconds. Removing the `unresolved` branch fails two of those tests
and removing `misfiled` fails a third.
Record-Id: r-ratchetdefects
Provenance: authored
Certainty: firm
Blast: system
Undo: easy
Ruled-out: always running the file unfiltered | it doubles the cost of every bound mutation to sharpen a diagnosis that only matters when the named test survived
Ruled-out: recording misfiled and unresolved in the baseline like the other gaps | the baseline records how far coverage reaches, and a registration that cannot be checked has no reach to record; carrying one would make the guard look accounted for
Ruled-out: parsing the test file for the registered name | describe blocks compose names at runtime, so a static read would reject valid registrations and accept a name that only appears to exist
Ruled-out: exporting the decision table from the ratchet itself | importing that script runs it, so the test would spawn sixty-six Vitest processes
Limit: the whole-file run attributes any failure to the mutation, which holds because the clean tree is green but would misread a flaky test as misfiled. That is a loud wrong answer rather than a silent one
Limit: this repairs how a mutation is read and repairs no guard. Thirteen exclusion-index properties remain uncovered and one scan remains genuinely inert, unchanged
Limit: nothing here checks that a registered claim matches what its test asserts. A mutation can be bound to the right test for the wrong property and this still reads bound
Verified: A/B recorded above, both directions, same registry; guard-mutations exits 0 on the clean tree with 64 bound, 1 inert, 1 unavailable, 13 uncovered — identical to before the change; injected unresolved and misfiled registrations both produce REGISTRATION DEFECT and exit 1; vitest 3744 passed, 13 skipped, 0 failed; tsc --noEmit clean on the root and bench tsconfigs
CommitLore — record lintTrailers: clean — 1 commit in Active constraints for the paths this PR touchesLimits (9)
Ruled out (11)
Trailer violations fail this check. Active constraints are informational — they are what the repository already decided, not a verdict on this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
vitest run <file> -t <name>exits 0 when the name matches nothing — it skipsevery test in the file and reports success. The ratchet read only that exit code,
so a registration naming a test that no longer exists was indistinguishable from
a mutation nothing reacted to. Both came back
inert.The A/B
Same registry, with one inert-baselined guard's
test_namechanged to a namethat does not resolve:
64 bound, 1 inert, 1 unavailable, 13 uncovered64 bound, 0 inert, 1 unresolved, 1 unavailableREGISTRATION DEFECTnaming the file and the missing nameThe old summary is character-for-character what a healthy run prints. Renaming a
test is routine, and nothing anywhere said the guard had stopped being checked.
The second gap already cost a session
In #852 four v6 guards were rejected and two were reported
inert. Applying thosemutations by hand failed a test both times — a different test than the registry
named. The ratchet's message says the property has no test defending it; what it
observes is that the named test did not fail. Those are the same output for an
uncovered guard and a misfiled one, and the misfiled one looks fine on inspection
because the mutation is real, the test is real, and the test does fail.
What the reading distinguishes now
misfiledandunresolvedare not coverage gaps, they are broken registrations,so
baseline.jsonmay not hold them and measuring one fails on sight. A recordedgap says how far coverage reaches and can be carried with a reason; a broken
registration says the recorded coverage cannot be checked at all.
Cost
The unfiltered second run happens only where the named test survived — two of
sixty-six here. Bound mutations, the common case, still cost one Vitest process.
Testing the runner
The ratchet spawns a process per mutation, so it cannot be exercised from inside
the suite it runs. The decision table moved to
scripts/guard-outcomes.mjsaspure functions;
test/guard-ratchet-outcomes.test.tscovers it in 2ms. Removingthe
unresolvedbranch fails two of those tests, removingmisfiledfails athird — verified in both directions.
Verification
This repairs how a mutation is read and repairs no guard: thirteen exclusion-index
properties remain uncovered and one scan remains genuinely inert. Nothing here
checks that a registered claim matches what its test asserts — a mutation can be
bound to the right test for the wrong property and this still reads
bound.