Skip to content

Tell a renamed test from an unbound guard in the mutation ratchet - #854

Merged
MongLong0214 merged 1 commit into
mainfrom
ratchet-registration-defects
Aug 24, 2026
Merged

Tell a renamed test from an unbound guard in the mutation ratchet#854
MongLong0214 merged 1 commit into
mainfrom
ratchet-registration-defects

Conversation

@MongLong0214

Copy link
Copy Markdown
Owner

vitest run <file> -t <name> exits 0 when the name matches nothing — it skips
every test in the file and reports success. The ratchet read only that exit code,
so a registration naming a test that no longer exists was indistinguishable from
a mutation nothing reacted to. Both came back inert.

The A/B

Same registry, with one inert-baselined guard's test_name changed to a name
that does not resolve:

old new
exit 0 1
summary 64 bound, 1 inert, 1 unavailable, 13 uncovered 64 bound, 0 inert, 1 unresolved, 1 unavailable
failures none REGISTRATION DEFECT naming the file and the missing name

The old summary is character-for-character what a healthy run prints. Renaming a
test is routine, and nothing anywhere said the guard had stopped being checked.

The second gap already cost a session

In #852 four v6 guards were rejected and two were reported inert. Applying those
mutations by hand failed a test both times — a different test than the registry
named. The ratchet's message says the property has no test defending it; what it
observes is that the named test did not fail. Those are the same output for an
uncovered guard and a misfiled one, and the misfiled one looks fine on inspection
because the mutation is real, the test is real, and the test does fail.

What the reading distinguishes now

bound        the registered test failed
misfiled     it passed, but the mutation failed some other test — named in the output
unresolved   the registered name matched nothing, so nothing was measured
inert        the mutation ran and no test in the file failed

misfiled and unresolved are not coverage gaps, they are broken registrations,
so baseline.json may not hold them and measuring one fails on sight. A recorded
gap says how far coverage reaches and can be carried with a reason; a broken
registration says the recorded coverage cannot be checked at all.

Cost

The unfiltered second run happens only where the named test survived — two of
sixty-six here. Bound mutations, the common case, still cost one Vitest process.

Testing the runner

The ratchet spawns a process per mutation, so it cannot be exercised from inside
the suite it runs. The decision table moved to scripts/guard-outcomes.mjs as
pure functions; test/guard-ratchet-outcomes.test.ts covers it in 2ms. Removing
the unresolved branch fails two of those tests, removing misfiled fails a
third — verified in both directions.

Verification

guard-mutations   exit 0 on a clean tree
                  64 bound, 0 misfiled, 0 unresolved, 1 inert, 1 unavailable, 13 uncovered
                  identical coverage to before the change
injected defects  both produce REGISTRATION DEFECT and exit 1
vitest run        3744 passed, 13 skipped, 0 failed
tsc --noEmit      clean on root and bench tsconfigs

This repairs how a mutation is read and repairs no guard: thirteen exclusion-index
properties remain uncovered and one scan remains genuinely inert. Nothing here
checks that a registered claim matches what its test asserts — a mutation can be
bound to the right test for the wrong property and this still reads bound.

…althy one

`vitest run <file> -t <name>` exits 0 when the name matches nothing. It skips
every test in the file and reports success. The ratchet read only that exit code,
so a registration pointing at a test that no longer exists was indistinguishable
from a mutation nothing reacted to, and both came back `inert`.

Renaming a test is routine. A/B on the same broken registry, with the test name
of an inert-baselined guard changed to one that does not resolve:

    old   exit 0   64 bound, 1 inert, 1 unavailable, 13 uncovered   no failures
    new   exit 1   64 bound, 0 inert, 1 unresolved, 1 unavailable   REGISTRATION DEFECT

The old summary is character-for-character what a healthy run prints. Nothing
anywhere said the guard had stopped being checked.

The second gap has already cost a session. Yesterday four v6 guards were rejected
and two were reported `inert`; applying those mutations by hand failed a test both
times, a different test than the registry named. The ratchet's message says the
property has no test defending it, but what it observes is that the *named* test
did not fail — the same output for an uncovered guard and for a misfiled one, and
the misfiled one looks fine on inspection because the mutation is real and the
test is real.

So the reading now distinguishes four things where it used to see two:

    bound        the registered test failed
    misfiled     it passed, but the mutation failed some other test — named
    unresolved   the registered name matched nothing, so nothing was measured
    inert        the mutation ran and no test in the file failed

`misfiled` and `unresolved` are not coverage gaps, they are broken registrations,
so `bench/cdeb/guards/baseline.json` may not hold them and measuring one fails on
sight. A recorded gap says how far coverage reaches and can be carried with a
reason; a broken registration says the recorded coverage cannot be checked at all,
and ratcheting that in defeats the baseline.

The unfiltered second run costs one extra Vitest process, and only on the path
where the named test survived — two of sixty-six here. Bound mutations, the
common case, still cost one.

The reading moved to `scripts/guard-outcomes.mjs` because the ratchet spawns a
Vitest process per mutation and cannot be exercised from inside the suite it
runs. The decision table is pure, so `test/guard-ratchet-outcomes.test.ts` covers
it in two milliseconds. Removing the `unresolved` branch fails two of those tests
and removing `misfiled` fails a third.

Record-Id: r-ratchetdefects
Provenance: authored
Certainty: firm
Blast: system
Undo: easy
Ruled-out: always running the file unfiltered | it doubles the cost of every bound mutation to sharpen a diagnosis that only matters when the named test survived
Ruled-out: recording misfiled and unresolved in the baseline like the other gaps | the baseline records how far coverage reaches, and a registration that cannot be checked has no reach to record; carrying one would make the guard look accounted for
Ruled-out: parsing the test file for the registered name | describe blocks compose names at runtime, so a static read would reject valid registrations and accept a name that only appears to exist
Ruled-out: exporting the decision table from the ratchet itself | importing that script runs it, so the test would spawn sixty-six Vitest processes
Limit: the whole-file run attributes any failure to the mutation, which holds because the clean tree is green but would misread a flaky test as misfiled. That is a loud wrong answer rather than a silent one
Limit: this repairs how a mutation is read and repairs no guard. Thirteen exclusion-index properties remain uncovered and one scan remains genuinely inert, unchanged
Limit: nothing here checks that a registered claim matches what its test asserts. A mutation can be bound to the right test for the wrong property and this still reads bound
Verified: A/B recorded above, both directions, same registry; guard-mutations exits 0 on the clean tree with 64 bound, 1 inert, 1 unavailable, 13 uncovered — identical to before the change; injected unresolved and misfiled registrations both produce REGISTRATION DEFECT and exit 1; vitest 3744 passed, 13 skipped, 0 failed; tsc --noEmit clean on the root and bench tsconfigs
@github-actions

Copy link
Copy Markdown

CommitLore — record lint

Trailers: clean — 1 commit in origin/main..f20857772c4ca22b69736995de8935a93552dee5
Active constraints: 9 limits · 11 ruled-out · 0 warnings — from 3 records over 3 changed paths

Active constraints for the paths this PR touches

Limits (9)

  • r-ratchetdefects f208577 — the whole-file run attributes any failure to the mutation, which holds because the clean tree is green but would misread a flaky test as misfiled. That is a loud wrong answer rather than a silent one
  • r-ratchetdefects f208577 — this repairs how a mutation is read and repairs no guard. Thirteen exclusion-index properties remain uncovered and one scan remains genuinely inert, unchanged
  • r-ratchetdefects f208577 — nothing here checks that a registered claim matches what its test asserts. A mutation can be bound to the right test for the wrong property and this still reads bound
  • r-guardratchet 26b1989 — the baseline is a floor. A guard can bind its claim against the one mutation recorded for it and still miss a different violation of the same claim
  • r-guardratchet 26b1989 — the reasons are written by the same author as the claims, so a gap reasoned narrowly can look more settled than it is
  • r-guardratchet 26b1989 — thirteen exclusion kinds remain uncovered and one scan inert; this records them and repairs neither
  • r-guardmutation 551921d — a mutation proves a guard reacts to one specific edit. A guard can bind its claim for that edit and miss a different violation of the same claim, so coverage here is a floor and not a proof
  • r-guardmutation 551921d — the claims were written by the same author as the guards, so a claim stated too narrowly produces a control that passes for a property nobody wanted
  • r-guardmutation 551921d — thirteen exclusion-index kinds remain inert; this change makes that visible and does not repair it

Ruled out (11)

  • r-ratchetdefects f208577 — always running the file unfiltered | it doubles the cost of every bound mutation to sharpen a diagnosis that only matters when the named test survived
  • r-ratchetdefects f208577 — recording misfiled and unresolved in the baseline like the other gaps | the baseline records how far coverage reaches, and a registration that cannot be checked has no reach to record; carrying one would make the guard look accounted for
  • r-ratchetdefects f208577 — parsing the test file for the registered name | describe blocks compose names at runtime, so a static read would reject valid registrations and accept a name that only appears to exist
  • r-ratchetdefects f208577 — exporting the decision table from the ratchet itself | importing that script runs it, so the test would spawn sixty-six Vitest processes
  • r-guardratchet 26b1989 — leaving the job in the gate while it is red | branch protection would block every merge until someone deleted the job, and deleting it removes the only thing that can see these gaps
  • r-guardratchet 26b1989 — allowing an improvement without updating the baseline | the record would drift below the measurement, and a baseline that overstates the gaps is as useless as one that understates them
  • r-guardratchet 26b1989 — recording gaps as a count instead of per property with a reason | a count cannot distinguish a control nobody wrote from one that cannot exist, and those need different work
  • r-guardmutation 551921d — fixing the inert guards in the same change | a runner that has never reported a real failure is not known to report one, and the red run is the evidence that it can
  • r-guardmutation 551921d — indexing guard functions instead of claims | that reproduces the exact failure this exists to stop, because the control comes back out of the implementation it is meant to test
  • r-guardmutation 551921d — treating an unexpressible control as a skip | it is indistinguishable from a control nobody attempted, and both were silently green before
  • r-guardmutation 551921d — folding this into the check job | it runs vitest once per mutation, so it belongs in its own job where its cost is visible

git log --follow accepts exactly one pathspec, so renames are not followed for 3 paths; query one path at a time to follow its rename chain

Trailer violations fail this check. Active constraints are informational — they are what the repository already decided, not a verdict on this PR.

@MongLong0214
MongLong0214 merged commit d48232f into main Aug 24, 2026
15 checks passed
@MongLong0214
MongLong0214 deleted the ratchet-registration-defects branch August 24, 2026 00:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant