Register the v6 buildability findings as tests, and fix four guards the ratchet rejected - #852
Merged
Merged
Conversation
The v6 findings had no test behind them. Nineteen were written, six guards
registered, and the ratchet rejected four of the six:
v6-regression-only-pass-is-not-a-functional-pass unavailable
v6-controls-judged-against-their-own-baseline unavailable
v6-bad-control-needs-two-agreeing-blind-judges inert
v6-floors-were-not-adjusted-to-fit-the-census inert
The two inert ones were not inert. Applying each mutation by hand and running
the suite failed a test both times — a different test than the registry named.
Changing the floor from 10 to 8 fails "holds the floors at the values the
preregistration fixed"; changing the verdict to GO fails "reads the shortfall
as TERMINAL_HOLD_FINAL rather than as a smaller study". I had registered each
against the other one's test.
So the ratchet was right that the registration was wrong and wrong about why.
Its message says the property has no test defending it; what it can actually
observe is that the named test did not fail. Those are the same output for a
guard nobody covered and a guard whose covering test is misfiled, and the second
is the one that looks fine on inspection — the mutation is real, the test is
real, the test does fail. Only the pairing is wrong, and the pairing is the part
no reviewer reads.
The two unavailable ones were literal: the ratchet requires a find string that
matches exactly once, and `"baseline_used": "per-candidate"` occurs 85 times in
the receipts and `"functional_pass": false` 13 times. Both now key on a single
receipt by candidate and variant, so the mutation lands on one row.
64 bound, 1 inert, 1 unavailable, 13 uncovered, 66 mutations run
The remaining inert and unavailable are the two that were already there.
Record-Id: r-v6ratchetmisfiled
Provenance: authored
Certainty: firm
Blast: local
Undo: easy
Ruled-out: reporting the four as covered and moving on | the ratchet's exit code was 1 and the guards were in baseline as bound, so this would have left four properties recorded as defended by tests that do not defend them
Ruled-out: relaxing the ratchet's exactly-one-match rule for JSONL | a find that matches 85 lines mutates whichever the replace hits first, which is not a control over anything in particular
Ruled-out: mutating the test file instead of the artifact | removing an assertion always makes a suite pass, so it measures nothing about whether the artifact is checked
Limit: the ratchet cannot distinguish "no test covers this" from "the wrong test is named" and this commit does not change that. It was caught by hand-applying two mutations, which is not something the pipeline does
Limit: 13 guards remain uncovered, all of them exclusion-index rows whose identities the census matches by value only. That was true before this commit
Verified: node scripts/guard-mutations.mjs exits 0 with no ratchet failures and all six v6 guards bound; tsc --noEmit clean on the root and bench tsconfigs; vitest 3733 passed, 13 skipped, 0 failed
CommitLore — record lintTrailers: clean — 1 commit in Active constraints for the paths this PR touchesLimits (32)
Ruled out (47)
Trailer violations fail this check. Active constraints are informational — they are what the repository already decided, not a verdict on this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The v6 buildability findings had no test behind them. This adds 19, registers six
properties in the mutation ratchet, and fixes four registrations the ratchet
rejected.
What the ratchet caught
Two guards were reported
inert— mutation applied, no test failed. Applyingeach by hand and running the suite failed a test both times, a different test
than the registry named:
"per_repository": 10→8"verdict": "TERMINAL_HOLD_FINAL"→"GO"Each was registered against the other's test. The ratchet was right that the
registration was wrong and wrong about why: its message says the property has no
test defending it, but what it observes is that the named test did not fail.
Those are the same output for an uncovered guard and for a guard whose covering
test is misfiled — and the second looks fine on inspection, because the mutation
is real, the test is real, and the test does fail. Only the pairing is wrong.
The other two were
unavailablefor a literal reason: the ratchet requires afind string matching exactly once, and
"baseline_used": "per-candidate"occurs85 times in the receipts,
"functional_pass": false13 times. Both now key on asingle receipt by candidate and variant.
The 19 tests
They read the committed artifacts, not fixtures, so they fail if the study
directory and the published result drift apart:
functional_pass = task AND regression, judged against per-candidate baselinesOne asserts 33 scouted candidates rather than 34: the candidate excluded for a
tree that revealed its own decision never reached a scout, so asserting 34 would
require having scouted past the firewall that stopped it.
Verification
The remaining inert and unavailable are the two that were already there. The
ratchet still cannot distinguish "no test covers this" from "the wrong test is
named"; this was caught by hand-applying two mutations, which is not something
the pipeline does.