CDEB-Fresh v8: the runner PR-B needs, and one arm that would have suppressed nothing - #859
Merged
Conversation
PR-B needs a runner that works on the seventeen, not on the fixture the smoke was written for. Building it surfaced a defect that would have quietly cost twenty episodes. v4-34aef026d81c2f6b has no Record-Id. Section 15's preflight had established its identity -- storage locator commit:f9a62917, ordinal 0, selecting exactly one of the 66 records its path scope returns -- but my section 14 freeze copied only the v7 decision block, which carries record_id: null. So the population the runner reads had nothing to suppress for that candidate. My first fallback then matched on a `storageLocator` field, and a record has no such field, so it removed nothing. An arm that removes nothing is an ON arm wearing a SUPPRESSED label. All ten repetitions would have reported a manipulation that never happened, and the pairs would have looked valid. suppression-identity.json now carries all seventeen and names the field it matches on -- the record's sha. It sits beside the population rather than inside it: the identity comes from section 15 rather than the section 14 import, and the population is hashed into the schedule seed, so adding a field would force a third re-freeze that buys nothing. dry-run-manipulation.py checks this against real trees. Not a pilot -- section 33 forbids that and this runs no coding agent and produces no outcome: it materialises each snapshot and asks the shipping build the question an episode would ask. 17/17 remove exactly the target with every other record surviving. That check matters more than it looks. Eight of the seventeen have record ids where one is a prefix of another -- `r-e0b001` and `r-e0b001b` are different decisions -- so a substring rule takes both while still reporting a plausible "1 removed". And the refusal is exercised rather than assumed: a target not in scope removes 0 and is refused; a null target removes 41, every record lacking an id, and is refused. run-episode.py now stops on any suppression that is not exactly one. Also here: section 20 allows one retry before a meaningful model turn and requires both rows preserved, which my round-C duplicate check would have crashed on. `itt_rows` separates the archive from the analysis -- both rows kept, exactly one counted, a superseded attempt with no successor refused. 20/20 mutations. batch.py walks the frozen schedule with one worker per repository, which satisfies section 18.4's three constraints without a scheduler, and refuses to start while measured_run_allowed is false.
…ery packet as leaking The measured judge packet, built at section 19's step 15 -- before the tree is destroyed at step 16, which is the only order that exists since the packet needs the tree. Inclusion is a whitelist of git-tracked files rather than a blacklist of known leaks. A blacklist is a list of the leaks somebody thought of, and anything the harness starts writing later arrives in the packet by default. `.git` stays out because branch names and commit messages name the arm, and the acceptance test stays out because it is the answer key. The cue scan needed narrowing, and the reason is worth writing down. Scanning the whole tree, a gitseed packet scored 71 hits on `Ruled-out:` and `Record-Id` -- all of them the repository's own documentation, because these are CommitLore-using repositories. Every packet would have come out cue-present, and section 23.9's cue-excluded sensitivity would have had nothing left to analyse. What can carry the arm is what differs between the arms: the diff and the files the agent touched. The base tree is identical in ON and SUPPRESSED by construction, so a marker sitting in a doc cannot tell a judge which arm it is reading. The scan is scoped there, and the base-tree marker count is kept as context rather than as a finding. Checked on two real trees rather than argued: an unchanged tree scores 0 cues against 71 base markers, and a line an agent might write naming its own arm scores 3, on arm-word-suppressed and experiment-assignment. Also moved: the packet record now sits beside the packet instead of inside it. A judge's directory should hold only what a judge is meant to read.
The second half of round C's claim-gate finding. Making evaluate_gate refuse an input with no stated origin was the first half; this is the thing that produces both the numbers and the origins, so a measured run has a derived dictionary rather than a typed one. gate-inputs.py reads rows and judgements off disk, attaches panel labels, runs the analysis, and returns each input beside the file or computation it came from. It refuses a partial seal outright: a gate answered on 200 of 340 rows is a different question from the one section 27 asks, and answering it anyway is how a partial run becomes a claim. It covers 13 of the 25 conditions. The other 12 need analysis output that does not exist before the measured run -- the bootstrap interval, the randomization p, RBDR, the completion and functional deltas, the judge and cue sensitivities -- or operator records like the two analysts matching. Writing "the gate is derived now" would be a stronger claim than the code supports, so preflight/gate-input-coverage.json names which 13 and which 12. The property that makes a partial derivation safe is that an underived input fails its condition rather than defaulting. Verified on a synthetic 340-row, 1,020- judgement seal: the gate failed on 15 conditions rather than passing on the 13 it could see.
The five prerequisites exist and are recorded with what each was verified against. measured_run_allowed stays false: that flag is PR-B's to flip, not this branch's.
CommitLore — record lintTrailers: clean — 16 commits in Active constraints for the paths this PR touchesLimits (12)
Ruled out (15)
Trailer violations fail this check. Active constraints are informational — they are what the repository already decided, not a verdict on this PR. |
…ined Section 24 asks for two independent analyses of the same sealed data. Exercised on synthetic data before any measured row exists: ANALYST-B was a fresh session in a different model family, given the frozen SAP text and the sealed rows and no reference implementation, and wrote its own analysis. ANALYST-A's numbers were computed and written down before B's answer was read. Eleven of twelve deterministic quantities matched. Panel label counts, the primary effect, both repository effects, both FVRs, all four reliability metrics and both completion rates -- two independent implementations of the same specification producing the same numbers, at the tolerances section 24 registers. The twelfth was RBDR, and B returned null: the SAP names it and gives no definition. It appears three times in the PRD -- once in the secondary outcomes list and twice as strong-claim thresholds -- and is defined nowhere. My implementation had invented `1 - FVR_on / FVR_suppressed`, a ratio of two aggregates that never looks at whether the same pair went both ways. Two of the 25 headline conditions were reading a quantity nobody had specified. Now registered (v8-d012): among pairs whose SUPPRESSED arm functionally revived, the fraction whose ON arm did not. That is what "blocked" means when the design pairs the same task and repetition across arms. Undefined when nothing revived, and the conditions fail rather than defaulting. A consequence worth stating rather than hiding: known_positive now reaches the claim in the synthetic simulation, where it was previously blocked. That was incidental -- the invented formula happened to fall below the 50% floor. The readiness test asserted "no scenario reaches the claim", which was true by accident; it now asserts what matters, that every no-effect and harmful scenario is blocked. Section 24's other correction (v8-d013): it required bootstrap quantiles and the permutation p to agree within 1e-6 between independent implementations. That is unachievable -- a quantile depends on which resamples were drawn, and independent code consumes the random stream differently from the same seed, so 1e-6 agreement would mean the two analysts wrote the same code. Measured: 2.8e-3 and 1.5e-3, ordinary Monte Carlo error at 2,000 replicates. Those two are now compared and their gap reported; everything deterministic is still compared strictly. 21/21 mutations, including two new ones on the pair-based RBDR.
ANALYST-B's code is kept, not just its answer. The claim being made is that a
second analyst implemented the plan without seeing the first, and two matching JSON
objects would look identical whether that happened or not. v6 lost its control
implementations by keeping only the prose account of them; the bytes are the part
that is not rebuildable from anything.
209 lines, its own structure -- a percentile helper, a per-pair AC1, its own Fleiss
kappa. Compared line by line against mine, one non-trivial line is shared:
return "PANEL_INDETERMINATE"
which is section 9.1's own literal. The two implementations share the
specification's vocabulary and nothing else.
That comparison is weaker evidence than it looks, and the artifact says so: two
people can write the same line independently. What it would have caught is the
other direction -- a high overlap would have been evidence against independence,
and there is none.
…hidden it Measuring the fixed per-episode cost before asking for sign-off turned up something worse than a cost. agent-operator-score's pristine snapshot runs 604 tests and 11 of them already fail. run-episode.py scored regression acceptance by exit code. So every one of that repository's 160 episodes would have failed regression whatever the agent did. P-DSFPS zero in both arms, a structural zero for half an equal-weight estimand, and section 27's "AOS P-DSFPS point effect > 0" unreachable by construction. gitseed's suite is green at baseline. That is what makes it nasty: the defect lives entirely in one repository, and half the data would have looked fine. Regression acceptance now means no failure outside a frozen baseline. regression-baseline.json records the expected failures per repository, measured on freshly materialised trees. The comparison is by test name rather than by count, because 11 against 11 is a pass even when one failure has been swapped for another. The baseline is frozen rather than recomputed per episode. Recomputing would let a flaky test join the baseline the first time it failed, which is the same as having no baseline -- so the first thing checked was whether it is flaky. Three runs on three freshly materialised trees: 11 not-ok lines, 10 distinct names, identical sets. One name genuinely appears in two places; nothing is intermittent. Verified in both directions. A pristine tree exits 1 and passes regression, because its failures are all expected. Appending a hard exit to a source file the suite asserts on fails regression and names the new failure.
A packet tree is millions of tokens if read whole, and no judge reads it whole. The calibration event streams show what actually happens: list the directory, read decision.txt, task.txt and diff.patch, then open the few source files the judge decides are relevant. That is not a defect -- the panel reached 43/47 unanimity under exactly this regime, so the judges were selected the way they will be measured. But it bounds what a judgement establishes. A violation visible only in a file no judge opened is scored compliant, and RESULT.md's limitations should say so rather than leaving "the judge saw the final tree" to be read as coverage. One difference between selection and measurement is recorded rather than smoothed over. Calibration packets were built by copying the tree; measured packets are built from `git ls-files`, so build output and untracked files stay out. Measured packets are narrower: 4.3 MB and 354 files for the largest, against a 15.5 MB median in calibration. The narrowing removes noise rather than evidence, since an agent's work lands in tracked files -- but it is a change in the instrument between the two phases and is not asserted to be neutral. Nothing here blocks the measured run.
…ant failure
The v6 tasks record `how_to_run` as an instruction for a person, and four of the
seventeen are not commands a shell can execute. One wraps its command in prose:
From the repository root: `node --test packages/schema/test/...`
Three name an interpreter that is not on PATH here -- two `pytest`, one `python`.
Each exits 127.
Scored by exit code, those four candidates fail task acceptance in every episode
whatever the agent does. Eighty episodes, four of seventeen candidates reduced to
structural zeros in both arms, and unlike the regression-baseline defect this one
spans both repositories, so no half of the data would have looked healthy.
acceptance-commands.json freezes a normalised command per candidate. The
normalisation is deliberately narrow -- take a backticked command out of its prose
wrapper, or replace a leading `pytest`/`python` with `python3 -m pytest`/`python3`
-- because anything broader is editing the task rather than making it runnable.
Four changed, thirteen untouched, and each change is recorded with its before and
after.
Every command was then verified on a freshly materialised tree with the acceptance
installed: 17/17 run, and 17/17 fail on the base tree. That second one is the
registered precondition -- a task whose acceptance already passes is not a task --
and it is re-established here rather than trusted from the v6 record.
run-episode.py now refuses an episode whose acceptance exits 126 or 127. That is an
infrastructure failure at evaluation time, not a verdict on the agent's tree, and
recording it as "acceptance failed" would record an outcome that was never measured.
This is the third defect of the same family found while building the runner: a
zero that looks like a result. The first was a suppression identity that removed
nothing; the second was regression scored by exit code against an already-red
suite.
Looking for a fourth defect of the family the last three belonged to -- a zero that reads as a result -- and there was one. `completed` was `exit_code == 0 and not timed_out`. That cannot distinguish a run that never reached a model from one that ran and produced a tree the acceptance rejects. A 401, a rate limit and a genuinely failing agent all exit non-zero. Section 20 turns on exactly that distinction: one retry is allowed before a meaningful model turn, and nothing may be replaced after one. A runner that cannot tell them apart cannot implement the rule, and an infrastructure failure recorded as a failed episode is a zero in the data for something never measured. The signal is the agent's own event stream rather than its exit code. A run that reached a model emits agent_message, command_execution or file_change; one that did not emits thread.started and turn.started and stops. Verified in both directions: a real episode's stream reports true, and a stub carrying thread.started, turn.started and a 401 reports false. The row now carries reached_a_model, pre_start_failure and retry eligibility. The runner still never retries on its own -- it records what section 20 needs, and itt_rows decides what is counted.
…scored Four defects found while building the runner all had one shape: a zero that reads as a result. Hunting them one at a time finds the ones you think of, so the next question was the general one -- can the scoring path produce a pass at all? It could not, on agent-operator-score. Applying the v7 rebuilt goodA patch, an implementation built to make the acceptance pass, scored regression_new=5 and functionally failed. Regression was being measured after the acceptance test was installed into the tree. `node --test` with no arguments discovers every test file, so installing the acceptance adds it to the regression suite -- and the effect is not one extra failure. Adding a file makes six census-style tests fail that pass without it. That defect survives the regression-baseline fix, because the baseline was measured on a tree with no acceptance installed. All 160 agent-operator-score episodes would have been functionally failed however well the agent did. No unit test on any individual step would have caught it. Each step is correct on its own; the defect is in their order. Only a check that asks for a positive outcome end to end can see it, which is why the question was worth asking as a question rather than as another hypothesis. Fixed: regression runs on the agent's tree first, then the acceptance is installed and run. Verified in both repositories -- base scores functional_pass false, goodA scores true. Recorded and not smoothed over: badA still scores four new regressions on agent-operator-score in the corrected order. That is a property of the control rather than the harness -- v6 built it to pass its own acceptance, not the suite -- and it is worth knowing before the run, because it suggests a functionally passing violation may be rarer than the design assumes.
The scoring-path control left a loose end worth pulling. Section 27 requires at least ten raw panel-violation events in the suppressed arm, and P-FVR counts only a violation that also passes functionally. One badA control had broken four tests while passing its own acceptance. If that were typical, a headline condition would be unreachable by construction rather than by result -- and that is the kind of thing to learn before 340 episodes, not after. Applying all seventeen badA controls and scoring them as episodes: every one passes its task acceptance, and thirteen introduce no new regression. Those thirteen are functionally passing violations. At ten repetitions each that is 130 of the 170 suppressed episodes on FVR-capable tasks, so ten events is reachable at a modest rate. Not structurally unreachable, which is what was being checked. The four that cannot are all agent-operator-score, and they are named. On those tasks the ruled-out approach also breaks the suite, so an agent taking it scores functionally failed rather than as a functionally passing violation. They can still contribute to P-DSFPS; they cannot contribute to P-FVR or RBDR. What this does not do is bound the measured rate. A control is one implementation of the ruled-out approach, written by v6, and an agent may take that approach in a way that does or does not break the suite. The question answered here is only whether the combination is possible at all. No coding agent ran and no outcome was produced.
… on the repository Five defects found while building the runner were all in the scoring path, which is where I had been looking. The delivery path had been checked for what SUPPRESSED removes and never for what ON delivers -- and if render() dropped the target, both arms would hand the agent the same nothing while the study still produced numbers. It does not. All seventeen ON payloads carry a Ruled-out, and all seventeen differ from their suppressed counterpart. The delivery path is sound. What the measurement also showed is worth keeping. On gitseed the target decision is one of three records in a payload of about a kilobyte. On agent-operator-score it is one of sixty in a payload of twenty-four. The same treatment differs by an order of magnitude in how salient it is, and the equal-weight estimand averages over that rather than adjusting for it. That is not a defect and nothing here corrects for it. It is recorded because it is a plausible source of a repository-level difference in effect, and reading one out of the results without this in view would attribute to the repository what may belong to the payload.
Every step of run-episode.py had been checked on its own, and five defects came out that way. But the two that would have cost the most lived between the steps -- regression measured after the acceptance was installed into the tree, and an arm that suppressed nothing. Nothing that tests a step can see either. So: the production run() on the first pair of the frozen schedule, both arms, with one substitution. Instead of invoking the coding agent it applies a v7 rebuilt goodA patch and writes an event stream of the shape the runner reads, so reached_a_model and first_mutation are exercised rather than bypassed. That substitution is what keeps it out of section 33's prohibition: no coding agent runs, no model is called, no outcome about the product is produced. What is produced is a row shaped exactly like a measured one, which is the only way to find out whether the row can be produced at all. 26/26. The row writes and reads back, regression scores against the frozen baseline before the acceptance is installed, the acceptance then passes, the packet is built before teardown and holds no .git and no judgement artifact, the tree is gone afterwards, and across the pair the two arms differ in delivered payload, in what was removed, and in packet id. Both arms, because the pair is the unit and a runner that works for ON and not for SUPPRESSED looks fine one episode at a time. One thing found on the way in: `started` held the wall-clock stamp and was then overwritten with a boolean from reached_a_model. It worked only because the elapsed time happened to be computed first. Renamed. What this does not say: that an agent will produce a passing tree, or how long an episode takes. It says the pipeline can carry an episode from a schedule row to a judge packet without losing or inventing anything.
The last untested path was the one that produces the study's primary measurement. The panel was calibrated on packets built by copying whole trees; the measured packets are narrower and built by different code, and nobody had checked that a judge can answer on one at all. It can. One frozen panel model, on a measured-style packet carrying the badA control for v4-0ecd7426eebc1cab, returned VIOLATION at high confidence with evidence paths inside tree/ -- matching the calibration key. And it reported its packet id as `work.judge-1`. That is the name of the scratch directory my own isolation fix had copied the packet into. A judge reads the id it is judging from where it is standing as readily as from packet_id.txt, and the old runner happened to stand in a directory named by the packet id. Every one of the 1,020 judgements would have carried the same wrong value, and nothing would have failed loudly -- linking is by directory, so the analysis would have run fine on a field that had quietly stopped meaning anything. Pulling that thread found its twin. run-episode.py wrote every packet to a directory literally named `packet`, and judge-run.sh derives the id from the directory it is handed. All 340 results would have collided in one folder. Neither component was wrong. The isolation isolated and the packet builder built. What was wrong was the context the two of them together put the judge in, and the only way to see it was to read what a judge actually said. Both now carry the packet id. Re-run returns the real one. The harness smoke asserts the directory name and is 28/28.
Verifying batch.py refuses while measured_run_allowed is false is safe, and it refuses with exit 2. I then verified the other half -- that it starts when the flag is true -- and it does, by beginning real episodes. Two assignments ran against the pinned model for about twenty seconds before a timeout killed them. The rollouts were read rather than guessed at. Both reached gpt-5.6-terra, with seventeen and twenty response items, and both modified zero files: the agents were still reading the tree. Nothing entered the study. No row was written for either assignment, so nothing reaches the analysis, and section 19 gives every episode a fresh worktree and a fresh HOME, so a killed run leaves no state a later run of the same assignment could inherit. Schedule, seed and population untouched. It is recorded anyway. Two of the 340 assignments have now had a first contact with the model outside the measured run. That does not violate fresh-session-per-episode and it is still the kind of fact that belongs in the record rather than in a deleted directory -- so the 105 MB of working trees are gone and incidents/2026-08-28-accidental-episode-start keeps the rollout digests, the line counts, the resolved model and the prompt hashes. The error was mine and it was a category error: testing "does it start" on a runner whose starting is the measured run. Only the refusal needed the real flag; the other half should have used the substitute agent that already exists for exactly this. batch.py now has --plan, which prints what would run and exits. Checking that it starts no longer starts it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
The pieces PR-B needs before it can run 340 episodes. No episode runs here and
measured_run_allowedstays false.run-episode.pybatch.pysuppression-identity.jsonepisode_packet.pygate-inputs.pyThe defect this surfaced
v4-34aef026d81c2f6bhas noRecord-Id. Section 15's preflight had established itsidentity — storage locator
commit:f9a62917, ordinal 0, selecting exactly one ofthe 66 records its path scope returns — but the section 14 freeze copied only the v7
decision block, which carries
record_id: null. So the population the runner readshad nothing to suppress for that candidate, and the first fallback I wrote matched
on a
storageLocatorfield that records do not have.An arm that removes nothing is an ON arm wearing a SUPPRESSED label. All ten
repetitions would have reported a manipulation that never happened, and the pairs
would have looked valid.
Checked against real trees, not argued
dry-run-manipulation.pymaterialises each of the seventeen snapshots and asks theshipping build the question an episode would ask. No coding agent runs and no
outcome is produced, so this is not the pilot section 33 forbids.
Those eight are why the check is not "one fewer record".
r-e0b001andr-e0b001bare different decisions, and a substring rule takes both while still reporting a
plausible count.
The refusal is exercised rather than assumed:
run-episode.pynow stops on any suppression that is not exactly one.Two smaller things
Section 20 allows one retry. My round-C duplicate check would have crashed on a
legitimate one.
itt_rowsseparates the archive from the analysis: both rows kept,exactly one counted, a superseded attempt with no successor refused.
The cue scan had to be narrowed. Scanning the whole tree, a gitseed packet
scored 71 hits on
Ruled-out:andRecord-Id— all of it the repository's owndocumentation, because these are CommitLore-using repositories. Every packet would
have come out cue-present and section 23.9's sensitivity would have had nothing to
analyse. The scan now covers the diff and the files the agent changed; the base tree
is identical across arms and cannot carry the assignment.
Not claimed
No measured episode has run.
gate-inputs.pycovers 13 of 25 conditions and theother 12 fail rather than default, so a partial derivation cannot reach the claim.