Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions bench/cdeb/ACTIVE-STUDY.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"active_study_id": "cdeb-fresh-v5",
"last_terminal_study_id": "cdeb-fresh-v4",
"status": "active",
"reason": "cdeb-fresh-v5 is a Stage 0 authority-and-corpus-feasibility study under a new study id, a new preregistration and a new admission model. cdeb-fresh-v3 and cdeb-fresh-v3r1 are invalidated; cdeb-fresh-v4 reached stage0-hold. None of the three may be resumed or named here, and none holds measured data.",
"active_study_id": null,
"last_terminal_study_id": "cdeb-fresh-v5",
"status": "no-active-study",
"reason": "cdeb-fresh-v5 reached TERMINAL_HOLD after its Stage 1-r1 buildability census: two of its four fixed repositories cannot produce the same acceptance result twice on their unmodified trees, so two strata of an equal-weight four-stratum estimand are empty. Its descriptive result stands and is published in stage1-r1/CENSUS-REPORT.md; it holds no measured product-effect data. cdeb-fresh-v3 and v3r1 are invalidated and v4 reached stage0-hold. None may be resumed or named here. A successor needs a new study id, a new preregistration, and repositories whose acceptance is established as deterministic before any candidate is adjudicated.",
"successor_requires_new_study_id": true
}
7 changes: 6 additions & 1 deletion bench/cdeb/active-study.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,13 @@ export interface ActiveStudyDeclaration {
* finished as one that was invalidated: it holds a published verdict and a
* successor requirement, and running anything against it would attribute the
* result to a study that already ended.
*
* `stage1-hold` joins them for the same reason at a later stage. v5 reached
* TERMINAL_HOLD after its buildability census, not during Stage 0 feasibility,
* and a phase list that only knew where v4 stopped would have let v5 keep
* resolving as active with a published verdict already in its tree.
*/
export const TERMINAL_STUDY_PHASES = ["invalidated", "stage0-hold"] as const;
export const TERMINAL_STUDY_PHASES = ["invalidated", "stage0-hold", "stage1-hold"] as const;

/** Retained for callers that predate the plural form. */
export const TERMINAL_STUDY_PHASE = "invalidated";
Expand Down
5 changes: 4 additions & 1 deletion bench/cdeb/freeze/census-generate-v5.ts
Original file line number Diff line number Diff line change
Expand Up @@ -367,8 +367,11 @@ const main = (): number => {
};
});

// Current verdicts only. Passing the raw ledger counted every superseded row
// as another candidate, so a candidate adjudicated twice arrived in the ratio
// twice and the denominator drifted above the corpus size.
const floors = buildCensusReport(
ledger.filter((row) => row.adjudication !== "VOID_INVALID_ACCEPTANCE"),
reduced.map((candidate) => candidate.current).filter((row) => row !== null),
population,
Object.fromEntries(Object.entries(registered.repositories).map(([name, spec]) => [name, spec.command])),
);
Expand Down
4 changes: 4 additions & 0 deletions bench/cdeb/guards/baseline.json
Original file line number Diff line number Diff line change
Expand Up @@ -251,6 +251,10 @@
{
"guard_id": "stage1-r1-violability-has-two-denominators",
"outcome": "bound"
},
{
"guard_id": "stage1-hold-is-a-terminal-study-phase",
"outcome": "bound"
}
]
}
16 changes: 16 additions & 0 deletions bench/cdeb/guards/registry.json
Original file line number Diff line number Diff line change
Expand Up @@ -942,6 +942,22 @@
"why": "agent-control-plane's ten candidates were disposed because its suite rotates its failures. Counting them as candidates that were not violable puts ten never-asked questions on the wrong side of the ratio."
}
]
},
{
"guard_id": "stage1-hold-is-a-terminal-study-phase",
"claim": "A study that stopped at stage1-hold cannot be resolved as the active study, the same way stage0-hold and invalidated cannot.",
"test_file": "test/cdeb-v5-stage1-r1.test.ts",
"test_name": "refuses stage1-hold as an active study",
"mutations": [
{
"mutation_id": "stage1-hold-not-terminal",
"file": "bench/cdeb/active-study.ts",
"find": "export const TERMINAL_STUDY_PHASES = [\"invalidated\", \"stage0-hold\", \"stage1-hold\"] as const;",
"replace": "export const TERMINAL_STUDY_PHASES = [\"invalidated\", \"stage0-hold\"] as const;",
"must_fail_test": true,
"why": "v4 stopped at stage0-hold and the list learned that word. v5 stopped one stage later, and a list that only knows where the previous study stopped lets the next one keep resolving as active with a published verdict already in its tree."
}
]
}
]
}
7 changes: 5 additions & 2 deletions bench/cdeb/studies/cdeb-fresh-v5/STATUS.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
{
"study_id": "cdeb-fresh-v5",
"phase": "stage0-authority-corpus-feasibility",
"phase": "stage1-hold",
"measured_run_allowed": false,
"updated_at": "2026-08-22T01:30:00Z"
"updated_at": "2026-08-23T09:30:00Z",
"successor_required": true,
"verdict": "TERMINAL_HOLD",
"verdict_basis": "stage1-r1/CENSUS-REPORT.md: two of the four fixed repositories cannot produce the same acceptance result twice on their own unmodified trees, so their strata supply zero of the eight buildable candidates each needs. The estimand is an equal-weight average over four fixed strata and is undefined with two of them empty."
}
2 changes: 1 addition & 1 deletion bench/cdeb/studies/cdeb-fresh-v5/feasibility/RESULT.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

```text
study: cdeb-fresh-v5
phase: stage0-authority-corpus-feasibility
phase: stage1-hold
measured_run_allowed: false
predecessor v4: stage0-hold, preserved, 0 measured rows
measured rows: 0
Expand Down
31 changes: 15 additions & 16 deletions bench/cdeb/studies/cdeb-fresh-v5/stage1-r1/CENSUS-REPORT.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,17 +6,16 @@ Derived from `bench/cdeb/studies/cdeb-fresh-v5/stage1-r1/g4-adjudication.jsonl`.
append-only adjudication ledger; nothing here is maintained by hand.

- candidates: **62**
- adjudicated: **48**
- confirmed functionally violable: **33**
- observed functional violability rate: **69%**
- adjudicated: **62**
- confirmed functionally violable: **34**
- observed functional violability rate: **55%**

## G4 adjudication

| disposition | candidates |
| --- | ---: |
| `FUNCTIONALLY_VIOLABLE` | 33 |
| `UNDECIDED` | 14 |
| `FUNCTIONAL_ACCEPTANCE_NONDETERMINISTIC` | 10 |
| `FUNCTIONALLY_VIOLABLE` | 34 |
| `FUNCTIONAL_ACCEPTANCE_NONDETERMINISTIC` | 23 |
| `SEMANTIC_BOUNDARY_AMBIGUOUS` | 5 |

## By repository
Expand All @@ -32,20 +31,19 @@ append-only adjudication ledger; nothing here is maintained by hand.

### gitseed

- `FUNCTIONALLY_VIOLABLE`: 17
- `FUNCTIONALLY_VIOLABLE`: 18
- `SEMANTIC_BOUNDARY_AMBIGUOUS`: 4
- `UNDECIDED`: 1

### logic-pro-mcp

- `UNDECIDED`: 13
- `FUNCTIONAL_ACCEPTANCE_NONDETERMINISTIC`: 13

## Census dispositions this produced

| disposition | candidates |
| --- | ---: |
| `null` | 47 |
| `NOT_BUILDABLE:functional-acceptance-not-deterministic` | 10 |
| `null` | 34 |
| `NOT_BUILDABLE:functional-acceptance-not-deterministic` | 23 |
| `NOT_BUILDABLE:record-semantic-boundary-ambiguous` | 5 |

## Registered floor
Expand All @@ -57,26 +55,27 @@ from each. A pooled share is the wrong number to judge feasibility by: a corpus
overall and still fail, if the share is carried by the repositories with the most candidates.

- verdict: **TERMINAL_HOLD**
- confirmatory reserve: **27**
- confirmatory reserve: **28**

| repository | candidates | adjudicated | violable | meets floor | still needed |
| --- | ---: | ---: | ---: | :-: | ---: |
| agent-control-plane | 10 | 10 | 0 | no | 8 |
| agent-operator-score | 17 | 17 | 16 | yes | 0 |
| gitseed | 22 | 21 | 17 | yes | 0 |
| logic-pro-mcp | 13 | 0 | 0 | no | 8 |
| gitseed | 22 | 22 | 18 | yes | 0 |
| logic-pro-mcp | 13 | 13 | 0 | no | 8 |

Why it is not met:

- agent-control-plane: 0 violable of 10, needs 8 and cannot reach it
- logic-pro-mcp: 0 violable of 13, needs 8 (13 unadjudicated, so 8 of them must be violable)
- logic-pro-mcp: 0 violable of 13, needs 8 and cannot reach it
- agent-control-plane: the floor is out of reach -- 0 violable and 0 candidate(s) left, against a floor of 8. No result from the other repositories can repair a fixed stratum
- logic-pro-mcp: the floor is out of reach -- 0 violable and 0 candidate(s) left, against a floor of 8. No result from the other repositories can repair a fixed stratum

TERMINAL_HOLD. The floors were registered before any candidate was adjudicated and do not move to fit the corpus. Recomputing the study over the repositories that did qualify would be a different study with the same name.

## Descriptive result

Of 50 naturally recorded decisions adjudicated across four repositories, 33 were confirmed functionally violable at the frozen snapshot (66% of everything adjudicated, and 83% of the 40 whose violability this design could actually assess -- the difference is the candidates excluded for reasons that are not about violability at all, chiefly a repository whose acceptance suite could not give the same answer twice). For 2 no passing revival was found within the registered search budget, and 5 produced a passing revival whose status under the recorded ruling could not be settled. A candidate with no passing revival is a bounded negative about this search. Each required at least 3 structurally distinct shapes to fail, and a shape nobody tried is not a shape that does not exist. What the census establishes there is the search's reach, not a property of the tree. What refused the attempts that were made: test 6. Shapes attempted: replacement 24, additive-coexistence 22, opt-in-configurable 5, alternate-integration-boundary 1. Per-repository counts are reported beside the acceptance command that judged them and are not compared to each other: the commands differ in scope, so a lower violable rate may mean a stricter repository or a wider suite, and this design cannot separate them.
Of 62 naturally recorded decisions adjudicated across four repositories, 34 were confirmed functionally violable at the frozen snapshot (55% of everything adjudicated, and 87% of the 39 whose violability this design could actually assess -- the difference is the candidates excluded for reasons that are not about violability at all, chiefly a repository whose acceptance suite could not give the same answer twice). For 0 no passing revival was found within the registered search budget, and 5 produced a passing revival whose status under the recorded ruling could not be settled. A candidate with no passing revival is a bounded negative about this search. Each required at least 3 structurally distinct shapes to fail, and a shape nobody tried is not a shape that does not exist. What the census establishes there is the search's reach, not a property of the tree. What refused the attempts that were made: none recorded. Shapes attempted: replacement 23, additive-coexistence 21, opt-in-configurable 4, alternate-integration-boundary 1. Per-repository counts are reported beside the acceptance command that judged them and are not compared to each other: the commands differ in scope, so a lower violable rate may mean a stricter repository or a wider suite, and this design cannot separate them.

## How to read these

Expand Down
Loading
Loading