Skip to content

Register the v6 buildability findings as tests, and fix four guards the ratchet rejected - #852

Merged
MongLong0214 merged 1 commit into
mainfrom
cdeb-v6-guard-registration
Aug 23, 2026
Merged

Register the v6 buildability findings as tests, and fix four guards the ratchet rejected#852
MongLong0214 merged 1 commit into
mainfrom
cdeb-v6-guard-registration

Conversation

@MongLong0214

Copy link
Copy Markdown
Owner

The v6 buildability findings had no test behind them. This adds 19, registers six
properties in the mutation ratchet, and fixes four registrations the ratchet
rejected.

What the ratchet caught

Two guards were reported inert — mutation applied, no test failed. Applying
each by hand and running the suite failed a test both times, a different test
than the registry named:

mutation registry named actually fails
"per_repository": 108 reads the shortfall as TERMINAL_HOLD_FINAL… holds the floors at the values the preregistration fixed
"verdict": "TERMINAL_HOLD_FINAL""GO" requires two blind judges to agree… reads the shortfall as TERMINAL_HOLD_FINAL…

Each was registered against the other's test. The ratchet was right that the
registration was wrong and wrong about why: its message says the property has no
test defending it, but what it observes is that the named test did not fail.
Those are the same output for an uncovered guard and for a guard whose covering
test is misfiled — and the second looks fine on inspection, because the mutation
is real, the test is real, and the test does fail. Only the pairing is wrong.

The other two were unavailable for a literal reason: the ratchet requires a
find string matching exactly once, and "baseline_used": "per-candidate" occurs
85 times in the receipts, "functional_pass": false 13 times. Both now key on a
single receipt by candidate and variant.

The 19 tests

They read the committed artifacts, not fixtures, so they fail if the study
directory and the published result drift apart:

  • predecessor is terminal, and the pool is exactly the 34 v5-prequalified rows
  • tasks were authored blind and frozen before any control was built
  • functional_pass = task AND regression, judged against per-candidate baselines
  • a Bad control needs two agreeing blind judges
  • the floors are the preregistered ones, and 17 < 22 reads as TERMINAL_HOLD_FINAL

One asserts 33 scouted candidates rather than 34: the candidate excluded for a
tree that revealed its own decision never reached a scout, so asserting 34 would
require having scouted past the firewall that stopped it.

Verification

node scripts/guard-mutations.mjs   exit 0, no ratchet failures
                                   64 bound, 1 inert, 1 unavailable, 13 uncovered
tsc --noEmit                       clean on root and bench tsconfigs
vitest run                         3733 passed, 13 skipped, 0 failed

The remaining inert and unavailable are the two that were already there. The
ratchet still cannot distinguish "no test covers this" from "the wrong test is
named"; this was caught by hand-applying two mutations, which is not something
the pipeline does.

The v6 findings had no test behind them. Nineteen were written, six guards
registered, and the ratchet rejected four of the six:

    v6-regression-only-pass-is-not-a-functional-pass   unavailable
    v6-controls-judged-against-their-own-baseline      unavailable
    v6-bad-control-needs-two-agreeing-blind-judges     inert
    v6-floors-were-not-adjusted-to-fit-the-census      inert

The two inert ones were not inert. Applying each mutation by hand and running
the suite failed a test both times — a different test than the registry named.
Changing the floor from 10 to 8 fails "holds the floors at the values the
preregistration fixed"; changing the verdict to GO fails "reads the shortfall
as TERMINAL_HOLD_FINAL rather than as a smaller study". I had registered each
against the other one's test.

So the ratchet was right that the registration was wrong and wrong about why.
Its message says the property has no test defending it; what it can actually
observe is that the named test did not fail. Those are the same output for a
guard nobody covered and a guard whose covering test is misfiled, and the second
is the one that looks fine on inspection — the mutation is real, the test is
real, the test does fail. Only the pairing is wrong, and the pairing is the part
no reviewer reads.

The two unavailable ones were literal: the ratchet requires a find string that
matches exactly once, and `"baseline_used": "per-candidate"` occurs 85 times in
the receipts and `"functional_pass": false` 13 times. Both now key on a single
receipt by candidate and variant, so the mutation lands on one row.

    64 bound, 1 inert, 1 unavailable, 13 uncovered, 66 mutations run

The remaining inert and unavailable are the two that were already there.

Record-Id: r-v6ratchetmisfiled
Provenance: authored
Certainty: firm
Blast: local
Undo: easy
Ruled-out: reporting the four as covered and moving on | the ratchet's exit code was 1 and the guards were in baseline as bound, so this would have left four properties recorded as defended by tests that do not defend them
Ruled-out: relaxing the ratchet's exactly-one-match rule for JSONL | a find that matches 85 lines mutates whichever the replace hits first, which is not a control over anything in particular
Ruled-out: mutating the test file instead of the artifact | removing an assertion always makes a suite pass, so it measures nothing about whether the artifact is checked
Limit: the ratchet cannot distinguish "no test covers this" from "the wrong test is named" and this commit does not change that. It was caught by hand-applying two mutations, which is not something the pipeline does
Limit: 13 guards remain uncovered, all of them exclusion-index rows whose identities the census matches by value only. That was true before this commit
Verified: node scripts/guard-mutations.mjs exits 0 with no ratchet failures and all six v6 guards bound; tsc --noEmit clean on the root and bench tsconfigs; vitest 3733 passed, 13 skipped, 0 failed
@github-actions

Copy link
Copy Markdown

CommitLore — record lint

Trailers: clean — 1 commit in origin/main..106a0fa9e96382e2de95aa8bae050d2d5b184ea6
Active constraints: 32 limits · 47 ruled-out · 0 warnings — from 14 records over 3 changed paths

Active constraints for the paths this PR touches

Limits (32)

  • r-v6ratchetmisfiled 106a0fa — the ratchet cannot distinguish "no test covers this" from "the wrong test is named" and this commit does not change that. It was caught by hand-applying two mutations, which is not something the pipeline does
  • r-v6ratchetmisfiled 106a0fa — 13 guards remain uncovered, all of them exclusion-index rows whose identities the census matches by value only. That was true before this commit
  • r-v5terminalphase 58d31c8 — the terminal-phase list still has to learn each new stopping word. A study that ends at a stage nobody has reached yet will resolve as active until someone notices, and the only defence is that the phase is written by the study itself
  • r-v5censusdenominators 890c3ee — the assessable denominator excludes candidates for reasons that are themselves study findings. A repository disqualified for instrument instability leaves the remaining corpus more homogeneous than the corpus that was sampled
  • r-v5censusdenominators 890c3ee — gitseed has one candidate outstanding and logic-pro-mcp has thirteen, so the descriptive rate will move
  • r-v5terminalhold e2581ab — twenty runs refute the determinism criterion but do not characterise the flakes. Seven tests seen once each gives a rate of roughly three runs in ten with at least one, and nothing here explains the mechanism
  • r-v5terminalhold e2581ab — the disposition is about this host. A different machine might not flake, and the study cannot distinguish "this suite is unstable" from "this suite is unstable here"
  • r-v5terminalhold e2581ab — the three surviving repositories are untouched by this and their numbers stand, but they now describe a corpus that cannot carry the confirmatory design
  • r-v5scopeconflict e26019b — the coverage check compares totals and skip counts. A patch that deletes one test and adds another keeps both numbers and passes
  • r-v5scopeconflict e26019b — the scope-conflict corroboration is a word match on the ruling -- a removal verb near a test noun. It confirms a claim already made rather than classifying rulings, but a ruling that describes removing coverage without those words would not corroborate
  • r-v5unchangedtreeisnotarevival 4721058 — telling an adjudicator that the repository's process documents do not bind it is a real intervention in what is being measured. It buys a technical answer to a technical question, and it means the census cannot speak to how much a project's own conventions deter a wrong path -- which is a genuinely interesting quantity this design now cannot see
  • r-v5unchangedtreeisnotarevival 4721058 — the unchanged-tree check catches an empty diff. An adjudicator that touches a file trivially and declines in substance passes it
  • r-v5acceptancereceipts ea67250 — two agreeing judgements measure stability, not correctness. A reading this design gets consistently wrong stays consistently wrong, and the rule is a floor rather than a warrant
  • r-v5acceptancereceipts ea67250 — the agreement rule was registered after six of twenty-eight pairs had been read. Its direction is conservative -- it can only move a candidate out of FUNCTIONALLY_VIOLABLE -- but it was not fixed before any of the data existed
  • r-v5acceptancereceipts ea67250 — a receipt proves the tree as it stands passes the registered command. Whether that tree implements one approach or several accumulated ones is a question the receipt cannot answer, and is left to the semantic judges who read the diff
  • r-v5guardnamestale 6202b5d — nothing prevents the next rename from doing this again. The ratchet catches it on the following CI run rather than at the moment of the rename, and that is the guarantee available
  • r-v5ssotreconciliation d3981ab — the 9.3 gate passes under an assumption the study cannot test. Nothing establishes the true between-candidate variance, the 12-candidate pilot cannot estimate one, and the corpus is fixed at 62 so no budget reduces it
  • r-v5ssotreconciliation d3981ab — the nine registered reasons are now the closed list, but no candidate has been disposed under any of them -- all 62 remain undecided pending oracle construction
  • r-v5stage1r1reviewfixes 87cacec — the pilot-gate custody gap is open and needs role separation -- execution, custody of arm-coded outcomes, and continuation held by different parties. That is an owner decision and it has to be settled before the pilot, not after
  • r-v5stage1r1reviewfixes 87cacec — the firewall now names who produced the maintenance need and refuses a producer not declared record-blind, but a declaration is not evidence. Making it evidence needs an attestable isolated authoring environment this layer does not have
  • r-v5stage1r1reviewfixes 87cacec — tau_squared_bound = 0.06 is a frozen assumption, not a measurement. It is conservative in the direction that matters -- lower true heterogeneity means the study detects more than promised -- but nothing here establishes the true value
  • r-v5stage1r1designlayer 28699d8 — the task-author firewall has still never run. Whoever built this corpus has read all 241 records, so the record-blind half cannot be satisfied from here -- it needs an author whose only inputs are the base tree and the maintenance need, with the manifest proving it
  • r-v5stage1r1designlayer 28699d8 — no oracle exists for any of the 62. Stage 0 recorded that reviewers thought one could be written; between that and a validated discriminating oracle sits the whole of G2, and nothing has crossed it
  • r-v5stage1r1designlayer 28699d8 — the between-candidate variance in the power table is a range I chose to bracket, not a measurement. The pilot supplies the real value, and only then does the detectable effect stop being a family of curves
  • r-v4qualification b8ff1b9 — G3 and G4 were judged from the commit message, the changed paths and the ruling. Neither reviewer read the current code or ran a test, so both are informed judgements about a maintenance task rather than measurements of one. G5 classifies whether an oracle could be written; none was built
  • r-v4qualification b8ff1b9 — this says nothing about whether recording decisions helps an agent. It says the four surveyed repositories cannot supply gold that is independent of the records being tested, which is a fact about these repositories and this gate
  • r-guardratchet 26b1989 — the baseline is a floor. A guard can bind its claim against the one mutation recorded for it and still miss a different violation of the same claim
  • r-guardratchet 26b1989 — the reasons are written by the same author as the claims, so a gap reasoned narrowly can look more settled than it is
  • r-guardratchet 26b1989 — thirteen exclusion kinds remain uncovered and one scan inert; this records them and repairs neither
  • r-guardmutation 551921d — a mutation proves a guard reacts to one specific edit. A guard can bind its claim for that edit and miss a different violation of the same claim, so coverage here is a floor and not a proof
  • r-guardmutation 551921d — the claims were written by the same author as the guards, so a claim stated too narrowly produces a control that passes for a property nobody wanted
  • r-guardmutation 551921d — thirteen exclusion-index kinds remain inert; this change makes that visible and does not repair it

Ruled out (47)

  • r-v6ratchetmisfiled 106a0fa — reporting the four as covered and moving on | the ratchet's exit code was 1 and the guards were in baseline as bound, so this would have left four properties recorded as defended by tests that do not defend them
  • r-v6ratchetmisfiled 106a0fa — relaxing the ratchet's exactly-one-match rule for JSONL | a find that matches 85 lines mutates whichever the replace hits first, which is not a control over anything in particular
  • r-v6ratchetmisfiled 106a0fa — mutating the test file instead of the artifact | removing an assertion always makes a suite pass, so it measures nothing about whether the artifact is checked
  • r-v5terminalphase 58d31c8 — leaving v5 as the active study since no successor exists | it holds a published TERMINAL_HOLD, and a study that can still be resolved as active is one that can still have work attributed to it
  • r-v5terminalphase 58d31c8 — inventing a v6 id to keep active_study_id populated | the resolver would point at a directory that does not exist, and a preregistration written to fill a field is not a preregistration
  • r-v5terminalphase 58d31c8 — relaxing the v4 test to assert only that no error is thrown | it would then pass whether or not v4 was active, which is the one thing it exists to check
  • r-v5censusdenominators 890c3ee — replacing the all-adjudicated rate with the assessable one | it is the number that favours the study, and a reader given only it cannot see that a quarter of the corpus was excluded by a broken instrument
  • r-v5censusdenominators 890c3ee — keeping only the all-adjudicated rate | it counts ten unasked questions as answers, and the answer it counts them as is the one this study would be accused of wanting to avoid
  • r-v5censusdenominators 890c3ee — closing the not-a-violation candidate as a bounded negative | a passing revival that does not violate the ruling means the search found working code rather than a wrong path, so the search has not finished and three distinct shapes have not failed
  • r-v5terminalhold e2581ab — a reproduction rule that counts a failure only when it repeats across two runs | the measured asymmetry would justify it -- noise only ever added failures, never removed one, in all twenty runs -- and it would cut contamination from 30 percent to about 2, which is exactly why it must not be introduced now: it is a change to the registered acceptance outcome, made after seeing the data, in the direction that keeps the study alive
  • r-v5terminalhold e2581ab — adding the seven rotating tests to the exclusion list | an exclusion chosen after watching which tests failed is the shape the receipt validator already refuses, and it would have grown the exclusions from nine to sixteen on evidence that each one flaked once
  • r-v5terminalhold e2581ab — waiting for the remaining 33 candidates before reporting a verdict | a fixed stratum at zero of eight with zero candidates left is arithmetic, not an open question, and reporting INCOMPLETE would invite work that cannot change the outcome
  • r-v5terminalhold e2581ab — declaring nondeterminism from the four earlier runs alone | those ran under uncontrolled and unrecorded load between 6 and 141, and the honest reading of them was that they could not be read
  • r-v5scopeconflict e26019b — allowing a revival to delete the test that blocks it and noting the deletion | a revival permitted to remove what fails it passes everything, and the census would then measure how freely adjudicators delete tests
  • r-v5scopeconflict e26019b — taking ACCEPTANCE_SCOPE_CONFLICT from the adjudicator's blocked_by field | it removes a candidate from the corpus, so the cheapest way to shrink a hard workload would be to call it unmeasurable
  • r-v5scopeconflict e26019b — dropping the category once its motivating candidate started producing patches | the argument does not depend on that candidate: any ruling whose content is the acceptance suite has the same problem, and finding it once means it will recur
  • r-v5unchangedtreeisnotarevival 4721058 — trusting the adjudicator's own implemented:false flag | it was accurate in all six cases and it is still a self-report, which is the class of evidence this whole layer replaced
  • r-v5unchangedtreeisnotarevival 4721058 — counting a declined attempt as a failed revival | the approach was never tried, so the run is evidence about the adjudicator and not about the tree, and folding it into the negatives would inflate them with refusals
  • r-v5unchangedtreeisnotarevival 4721058 — leaving the prompt alone and treating policy refusals as a real finding | the finding would be "an agent would not do it", and no reader of the census would take the number that way
  • r-v5acceptancereceipts ea67250 — keeping the thirty prose-backed verdicts on the grounds that only agent-control-plane had a blocked sandbox | the objection is not that those runs went wrong, it is that nothing in the artifact can tell, and a rule that trusts prose wherever it has no reason to doubt it is the rule that produced the seven
  • r-v5acceptancereceipts ea67250 — discarding the thirty and re-running the search | the search is the expensive half and it already happened; the patched trees are still on disk, so acceptance can be replayed mechanically and the receipt built from what it actually produces
  • r-v5acceptancereceipts ea67250 — taking a single blind semantic judgement per revival | one judgement records whichever run happened to be kept, and one of the first six pairs disagreed with itself
  • r-v5acceptancereceipts ea67250 — editing the historical TREE_ENFORCED rows to the new name | an overturned negative is evidence that the search budget was once too small, and a rewritten row cannot show that
  • r-v5guardnamestale 6202b5d — adding a static check that every registry test_name appears in its file | test.each generates names at run time, so the check reports a false positive on a guard that genuinely binds, and a check that cries wolf gets muted
  • r-v5ssotreconciliation d3981ab — keeping my 15pp/1,650-episode envelope | it was derived by inverting a detectable-effect formula against a variance the pilot was supposed to supply, which is the sizing direction section 9 exists to remove
  • r-v5ssotreconciliation d3981ab — running the 9.3 gate with my more conservative variance assumption | that substitutes my judgement for the owner's registered method and would have produced a HOLD the SSOT does not call for
  • r-v5ssotreconciliation d3981ab — reporting the 9.3 PASS without the heterogeneity table | the registered simulation assumes candidates benefit equally, and a reader who cannot see what that assumption buys cannot judge a null result
  • r-v5ssotreconciliation d3981ab — deleting the analytic sizing path now that it is not the gate | it is the more pessimistic of the two and a disagreement between them is worth seeing rather than averaging
  • r-v5stage1r1reviewfixes 87cacec — treating the reviewer's findings as claims to weigh | six of them were statements about what the code does, and running the code settled each one in under a minute
  • r-v5stage1r1reviewfixes 87cacec — keeping 8 repeats and reporting the detectable effect as a range | the range was over a parameter the design had no way to obtain, which is how an operator ends up choosing the favourable end of it
  • r-v5stage1r1reviewfixes 87cacec — lowering the minimum important effect to what 8 repeats reaches | that is the move the HOLD rule exists to refuse, and writing it into the rule that refuses it would have been circular
  • r-v5stage1r1reviewfixes 87cacec — claiming the pilot custody finding was closed by the schema | the record now cannot carry an arm contrast and the operator can still have watched the runs; a control over bytes is not a control over people
  • r-v5stage1r1designlayer 28699d8 — filling the runtime lock with plausible placeholder values | an unpinned runtime that reads as pinned is worse than an empty lock, because the next reader stops asking
  • r-v5stage1r1designlayer 28699d8 — marking the 62 screen-surviving candidates BUILDABLE | the screens can only refute, and calling a candidate buildable without an oracle is the claim the census exists to check
  • r-v5stage1r1designlayer 28699d8 — patching the six defects into the failed Stage 1 draft | its own section 7 says anything but the deferred N makes a change a new preregistration, so amending the document that defines amendment is the failure it guards
  • r-v5stage1r1designlayer 28699d8 — reporting PASS on the eleven satisfied criteria | four unresolved P0/P1 is a HOLD under section 19, and a partial pass reads as readiness to whoever approves execution
  • r-v4qualification b8ff1b9 — adjudicating the 92 split gates myself | the study operator reading their own corpus, already knowing how the pair voted, is the least blind reader available; a third blind vote costs one more session and is a vote rather than an override
  • r-v4qualification b8ff1b9 — averaging or passing an unresolved disagreement | it would put a candidate in the corpus that no two reviewers agreed on, and the disagreement would stop being visible
  • r-v4qualification b8ff1b9 — relaxing the quote-correspondence floor after seeing 8% | the floor was fixed in code and in the deviation record before any overlap was computed, and moving it now would let the count choose the method
  • r-v4qualification b8ff1b9 — accepting any rejection found in the same commit | it qualifies candidate X on evidence about decision Y, which is how a corpus fills up without meaning anything
  • r-guardratchet 26b1989 — leaving the job in the gate while it is red | branch protection would block every merge until someone deleted the job, and deleting it removes the only thing that can see these gaps
  • r-guardratchet 26b1989 — allowing an improvement without updating the baseline | the record would drift below the measurement, and a baseline that overstates the gaps is as useless as one that understates them
  • r-guardratchet 26b1989 — recording gaps as a count instead of per property with a reason | a count cannot distinguish a control nobody wrote from one that cannot exist, and those need different work
  • r-guardmutation 551921d — fixing the inert guards in the same change | a runner that has never reported a real failure is not known to report one, and the red run is the evidence that it can
  • r-guardmutation 551921d — indexing guard functions instead of claims | that reproduces the exact failure this exists to stop, because the control comes back out of the implementation it is meant to test
  • r-guardmutation 551921d — treating an unexpressible control as a skip | it is indistinguishable from a control nobody attempted, and both were silently green before
  • r-guardmutation 551921d — folding this into the check job | it runs vitest once per mutation, so it belongs in its own job where its cost is visible

git log --follow accepts exactly one pathspec, so renames are not followed for 3 paths; query one path at a time to follow its rename chain

Trailer violations fail this check. Active constraints are informational — they are what the repository already decided, not a verdict on this PR.

@MongLong0214
MongLong0214 merged commit cfb2552 into main Aug 23, 2026
15 checks passed
@MongLong0214
MongLong0214 deleted the cdeb-v6-guard-registration branch August 23, 2026 23:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant