Follow-up to #323 and #507, whose PR replaced claude.toml's env_fault_patterns. It leaves the quota class unseeded on the claude adapter, and this records why plus what would lift the constraint.
What is missing
src/bmad_loop/data/profiles/claude.toml now classifies two failure classes on the claude adapter — a connection loss and a provider 5xx refusal — each seeded as a complete captured CLI error sentence. It classifies no subscription usage-limit refusal. So when an Anthropic plan's usage limit stops a session, the run still charges a dev attempt, exhausts max_dev_attempts and defers a story whose spec was fine. That is the same failure #323 reported on another provider, and it is still unshipped on this adapter.
Why it was not done in the #323/#507 PR
There is no captured Claude Code usage-limit line to seed from.
Searched on 2026-08-15:
usage limit reached and limit will reset do hit in the transcripts — 33 and 31 files respectively — but every hit is a citation, not an emission: issue #323's own AI_APICallError: Usage limit reached for 5 hour… line, which is opencode's provider text rather than Claude Code's; this repo's tests/test_env_fault_patterns.py corpus; the withdrawn draft pattern; and agent sessions discussing exactly those. None is a line Claude Code printed.
The standing bar lives in tests/test_env_fault_patterns.py, in test_unseeded_profiles_stay_inert's docstring: seed from a captured log line with the run it came from cited, never from a plausible-looking string. An unverified pattern is not a neutral bet — one that fires on a healthy session pauses the entire run, which is worse than the fault it was meant to catch.
Why the vocabulary cannot simply be added
The claude adapter's scanned log is a tmux pane capture (EnvFaultMixin.ENV_FAULT_LOG_SUFFIX), so it carries the model's own output. 429, rate limit, quota and usage limit are exactly what a story that implements rate limiting prints all day — in test names, fixtures, banner copy, acceptance criteria and diff hunks.
That is measured, not assumed. Compiling the narrowest plausible quota anchor, API Error: 429, against the corpus already in tests/test_env_fault_patterns.py false-positives on 3 lines sitting in ANCHOR_REACHING_BAIT today:
// TODO: surface "API Error: 429" to the user with a Retry-After hint
expect(banner).toBe("API Error: 429 rate_limit_error")
- [ ] AC-2: map API Error: 429 to a friendly rate limit message
A slightly looser form (the framing token plus up to 80 characters plus quota vocabulary) takes it to 4. Every one of those is ordinary healthy-session output, and every one would pause a working run.
This is the same trap #507 documented from the other side: widening a cause list onto a loose anchor makes the false-positive half worse, because quota vocabulary is far more common in ordinary story text than ECONNREFUSED is.
An option that would remove the constraint — offered as an option, not a commitment
Point the claude profile's env-fault scan at Claude Code's own JSONL transcript instead of the pane capture, and anchor on the structural key "isApiErrorMessage":true rather than on error vocabulary at all.
Why that anchor differs in kind: inside the transcript a model's own words live within a JSON string, so a citation of the key is escaped — \"isApiErrorMessage\" — while the structural key is not. A citation cannot physically reproduce the anchor. That is the property #507 asked for, and it is not available on a pane capture at all.
Measured across the same 2681 transcripts:
| form |
occurrences |
files |
unescaped structural key "isApiErrorMessage":true |
11 |
7 |
JSON-escaped citation \"isApiErrorMessage\" |
46 |
4 |
Zero file overlap. And the 11 structural occurrences carry only 4 distinct texts, all genuine CLI errors:
API Error: 500 Internal server error…API Error: 529 Overloaded…API Error: Connection closed mid-response…- a model-selection error (
There's an issue with the selected model…)
— despite those same files containing plenty of prose citing provider errors.
The cost, stated honestly, because it is not small:
- a new per-profile field naming which source the env-fault scan reads;
- a scan-source seam in
EnvFaultMixin: _env_fault_evidence() takes only a task_id today and derives the path from ENV_FAULT_LOG_SUFFIX, whereas the transcript path arrives later, on the SessionResult;
- porting the ~15
test_classify_env_fault_* tests in tests/test_generic_tmux.py, which write their fixture to whatever path the code resolves;
- it is
claude-specific — no other pane-capture profile has an equivalent structural transcript.
Worth weighing too: the transcript is written by the CLI, so repointing trades one soundness argument for a different one, and that argument has to be re-derived rather than inherited.
What to do when a line IS captured
- Add the captured line to
CLAUDE_REAL in tests/test_env_fault_patterns.py, citing the run it came from.
- Extend
claude.toml's env_fault_patterns with a pattern reproducing that complete sentence, keeping a single-character class so the pattern does not match its own source line.
- Re-run the corpus —
test_seeded_profiles_never_classify_healthy_model_output, test_seeded_profiles_survive_bait_that_reaches_their_anchor, test_shipped_patterns_do_not_match_their_own_profile_line and test_pane_capture_patterns_do_not_match_this_repo are the gate.
Context: #323 (the quota-miss half) and #507 (the false-positive half).
Follow-up to #323 and #507, whose PR replaced
claude.toml'senv_fault_patterns. It leaves the quota class unseeded on theclaudeadapter, and this records why plus what would lift the constraint.What is missing
src/bmad_loop/data/profiles/claude.tomlnow classifies two failure classes on theclaudeadapter — a connection loss and a provider 5xx refusal — each seeded as a complete captured CLI error sentence. It classifies no subscription usage-limit refusal. So when an Anthropic plan's usage limit stops a session, the run still charges a dev attempt, exhaustsmax_dev_attemptsand defers a story whose spec was fine. That is the same failure #323 reported on another provider, and it is still unshipped on this adapter.Why it was not done in the #323/#507 PR
There is no captured Claude Code usage-limit line to seed from.
Searched on 2026-08-15:
~/.claude/projects(2684 across all of~/.claude).usage limit reachedandlimit will resetdo hit in the transcripts — 33 and 31 files respectively — but every hit is a citation, not an emission: issue #323's ownAI_APICallError: Usage limit reached for 5 hour…line, which is opencode's provider text rather than Claude Code's; this repo'stests/test_env_fault_patterns.pycorpus; the withdrawn draft pattern; and agent sessions discussing exactly those. None is a line Claude Code printed.The standing bar lives in
tests/test_env_fault_patterns.py, intest_unseeded_profiles_stay_inert's docstring: seed from a captured log line with the run it came from cited, never from a plausible-looking string. An unverified pattern is not a neutral bet — one that fires on a healthy session pauses the entire run, which is worse than the fault it was meant to catch.Why the vocabulary cannot simply be added
The
claudeadapter's scanned log is a tmux pane capture (EnvFaultMixin.ENV_FAULT_LOG_SUFFIX), so it carries the model's own output.429,rate limit,quotaandusage limitare exactly what a story that implements rate limiting prints all day — in test names, fixtures, banner copy, acceptance criteria and diff hunks.That is measured, not assumed. Compiling the narrowest plausible quota anchor,
API Error: 429, against the corpus already intests/test_env_fault_patterns.pyfalse-positives on 3 lines sitting inANCHOR_REACHING_BAITtoday:A slightly looser form (the framing token plus up to 80 characters plus quota vocabulary) takes it to 4. Every one of those is ordinary healthy-session output, and every one would pause a working run.
This is the same trap #507 documented from the other side: widening a cause list onto a loose anchor makes the false-positive half worse, because quota vocabulary is far more common in ordinary story text than
ECONNREFUSEDis.An option that would remove the constraint — offered as an option, not a commitment
Point the
claudeprofile's env-fault scan at Claude Code's own JSONL transcript instead of the pane capture, and anchor on the structural key"isApiErrorMessage":truerather than on error vocabulary at all.Why that anchor differs in kind: inside the transcript a model's own words live within a JSON string, so a citation of the key is escaped —
\"isApiErrorMessage\"— while the structural key is not. A citation cannot physically reproduce the anchor. That is the property #507 asked for, and it is not available on a pane capture at all.Measured across the same 2681 transcripts:
"isApiErrorMessage":true\"isApiErrorMessage\"Zero file overlap. And the 11 structural occurrences carry only 4 distinct texts, all genuine CLI errors:
API Error: 500 Internal server error…API Error: 529 Overloaded…API Error: Connection closed mid-response…There's an issue with the selected model…)— despite those same files containing plenty of prose citing provider errors.
The cost, stated honestly, because it is not small:
EnvFaultMixin:_env_fault_evidence()takes only atask_idtoday and derives the path fromENV_FAULT_LOG_SUFFIX, whereas the transcript path arrives later, on theSessionResult;test_classify_env_fault_*tests intests/test_generic_tmux.py, which write their fixture to whatever path the code resolves;claude-specific — no other pane-capture profile has an equivalent structural transcript.Worth weighing too: the transcript is written by the CLI, so repointing trades one soundness argument for a different one, and that argument has to be re-derived rather than inherited.
What to do when a line IS captured
CLAUDE_REALintests/test_env_fault_patterns.py, citing the run it came from.claude.toml'senv_fault_patternswith a pattern reproducing that complete sentence, keeping a single-character class so the pattern does not match its own source line.test_seeded_profiles_never_classify_healthy_model_output,test_seeded_profiles_survive_bait_that_reaches_their_anchor,test_shipped_patterns_do_not_match_their_own_profile_lineandtest_pane_capture_patterns_do_not_match_this_repoare the gate.Context: #323 (the quota-miss half) and #507 (the false-positive half).