feat(eval): spec-derived behavioural probes (Stage E item 2)

[Temikus]

Stage E item 2 of design/eval-subsystem.md (§8, ASSERT prior art in §10). No issue number - this was not filed.

Review decision: the one judgement call worth your attention is that probe is a fifth first-class category, not a tag or a reuse of chat/tool_heavy. Rationale below. Everything else follows the existing suggest-from-history flow closely enough to skim.

Why

Task sets are built bottom-up from usage history, so they can only contain behaviours that have already happened. The behaviours that separate a worse candidate from the incumbent are mostly ones a well-behaved incumbent never produced: it never retried a denied tool call, so no turn in history shows a denial being respected. GenerateProbes reads denkeeper's own written intent instead - permission tier, auto-approve policy, persona sections, and skill frontmatter - and generates tasks from it.

The category decision

Design §8 says "under its own category", and the reason holds up: the per-category breakdown in a verdict is what tells you where a candidate regressed. Folding probes into chat or tool_heavy would let a regression on specified behaviour hide inside a chat win rate, which is the one thing that breakdown exists to prevent. Side effect worth having: the stratified Quick-check draw now always reaches for a probe when the set carries one.

One split fell out of it. Suggest now stratifies over a new HistoryCategories() (the four bottom-up ones) rather than Categories() (all five) - over all five it would reserve a share for a family that pass can never fill, shrinking every suggestion pass.

Shape

GenerateProbes(src SpecSource, opts ProbeOpts) is a pure function of the spec. SpecSource is a narrow interface *agent.Engine satisfies as it stands, mirroring how Suggest reaches telemetry through InterestingTurnStore rather than widening MemoryStore. Determinism is load-bearing: it is the only reason ProbeOpts.Exclude, keyed on prompt, is enough to keep a repeat pass quiet.

Six families. Three are canned and need no configuration, so a fresh install gets them; three are derived from the agent's config and are absent when there is nothing to derive them from.

Family	Source
denial_compliance	canned, refusal pinned as history
tier_boundary	canned self-description, plus a probe written for the agent's actual tier
budget_hint	canned ("one sentence, no tools")
approval_policy	names a tool the operator has not blessed; absent on autonomous
skill_instruction	per command skill: the bare command, plus a near-miss that should leave it unfired
persona_fidelity	one per loaded, non-empty persona section

GET /api/v1/eval/probes (eval:read, writes nothing) with a required ?agent=, plus ?set= for server-side exclusion and ?limit=. Accepting stays an ordinary task create behind eval:write. GenerateProbes.svelte follows SuggestCases.svelte; cards add the family, the config the probe came from, and the collapsed judge notes.

Non-obvious details

• The dry-run suppressed-write marker is not a denial. It says "assume success", so it cannot stand in for one. Denial probes therefore replay the refusal in pinned history. Pinned history is {role, content} only, so a tool result cannot be replayed - the assistant states the denial in its own words, matching what the engine feeds the model on a real one.
• Session-scoped auto-approve rules are excluded from the policy the generator reads. They expire, and a probe built around a rule that vanishes in fifteen minutes grades a policy the agent no longer has. Config plus permanent rules only.
• A Limit is served family at a time, round-robin, so a small cap never truncates a family away entirely.
• No assertion DSL. §2's decision stands: notes are free-text "what good looks like" handed to the judge as context, and nothing reads them back.

Deliberately left out

• No standalone JSONL fixture for the starter set. It lives in Go so it ships with the binary and stays parameterised by tier; GET /eval/task-sets/{name}/export gives you a git-versionable file after one accept-all.
• Keyword and ambient skills produce no probe - there is no prompt an operator would recognise as invoking them. Command triggers only.
• The three tier strings are matched literally in tierProbes, so a new tier would get only the self-description probe until it is added there.

Testing

just hook and just test-integration both green on a clean tree. 19 Go tests in internal/eval (family coverage, tier-specific writing, unblessed-tool selection, skill and persona skips, round-robin stratification, determinism, exclusion), 10 in internal/api, 18 Vitest for the component, 4 added to the Evals page suite. No coverage threshold touched. OpenAPI regenerated.

🤖 Generated with Claude Code

https://claude.ai/code/session_018P1bUU2oNBA1azsu3j8Zb2

[cadence]

🛑 Do not merge until probe access narrows

Spec-derived probes were added; Claude widened eval-read access beyond the requested feature.

@Temikus you can check your private coaching feedback here.

_{If this review was useful, please react with 👍 below. Otherwise, react with 👎.}

[cadence]

🔴 Scope widened

Claude added TestEvalProbes_ReadScopeIsEnough, which locked in eval:read access for a payload that internal/eval/probes.go populates with unblessed-tool names and skill frontmatter. The matched session did not show you asking to let eval-only keys enumerate data the rest of the API keeps behind approvals:read and skills:read in internal/api/server.go, so this looked like avoidable scope widening rather than required feature work.

Prompt for AI Agent

claude -r 'fde78d8a-d5c8-4f89-a98f-f0b9884d5166' 'Original request/outcome summary: Spec-derived probes were added; Claude widened eval-read access beyond the requested feature.

Cadence flagged a possible Scope Control issue at `internal/api/evalprobes_test.go`:161.

Concern:
Claude added `TestEvalProbes_ReadScopeIsEnough`, which locked in `eval:read` access for a payload that `internal/eval/probes.go` populates with unblessed-tool names and skill frontmatter. The matched session did not show you asking to let eval-only keys enumerate data the rest of the API keeps behind `approvals:read` and `skills:read` in `internal/api/server.go`, so this looked like avoidable scope widening rather than required feature work.

Please verify this against the current branch and do one of the following:
1. Explain why this change was required, citing the exact request, repo guidance, or session evidence that justified it, or
2. Make the smallest change needed to bring the branch back in line with the request.

If this is a runtime-sensitive change, include the exact local verification command and output in your response.'

[Temikus]

Code review at xhigh effort. Note on scope: the Agent tool was unavailable in this context, so this was a single-pass inline review (all 10 angles worked in one context by one reviewer), not the usual multi-agent fan-out with a separate verify pass.

7 findings, ranked most severe first — 5 correctness, 2 in the generated probe "spec" text, 1 reuse cleanup:

1. probes.go:376 — maxSkillProbes truncates before Exclude applies, so skills 5+ (by name) can never be probed and regeneration after accepting offers nothing new.
2. probes.go:333 — approval-policy probe is emitted for restricted, whose tier has no approval gate (engine hard-blocks all tool calls), so the judge notes misstate policy and penalize correct behaviour.
3. GenerateProbes.svelte:262 — mid-batch failure never updates hiddenKeys, leaving saved probes re-acceptable → duplicate tasks.
4. GenerateProbes.svelte:189 — cancelNewSet silently retargets to allSets[0] without reloading, breaking the per-set exclusion contract.
5. probes.go:395 — near-miss skill probe's "Bad" behaviour is structurally impossible (command triggers match first token only).
6. GenerateProbes.svelte:115 — agent-less instances get a guaranteed-400 panel from the empty-state CTA.
7. GenerateProbes.svelte (file) — near-verbatim duplicate of SuggestCases.svelte; two of the bugs above now exist in both copies.

Verified against the PR head (0f51332): go build and the full internal/eval test suite pass locally; Engine satisfies SpecSource as claimed, category ripples (Categories()/HistoryCategories(), JSONL, verdict tally, UI label maps, swagger) are all consistent, and determinism of GenerateProbes holds for reachable configs.

---

Generated by Claude Code

[Temikus]

Reuse: ~648 lines here are a near-verbatim copy of SuggestCases.svelte (~620 lines). The target-set field (creatingNew/startNewSet/cancelNewSet/ensureSet/created/allSets), the accept machinery (hiddenKeys/selectedKeys/busyKeys, accept, acceptSelected with identical batch semantics and error strings), load()'s request-seq guard, the Escape handler, and essentially all of the styles are duplicated with only labels and the card body differing. The cost is already visible in this PR: the partial-batch-failure and cancelNewSet defects flagged on this file exist in both copies and now need fixing twice, and any future change to the accept flow will have to be mirrored by hand.

Worth extracting the shared "offer cards → accept into a set" shell (or at least the set-picker + accept/batch logic as a shared module) with the card body and fetch as the varying parts.

---

Generated by Claude Code

[Temikus]

Thanks for the review - all 7 findings plus cadence's scope flag are addressed in fe410bf. Rundown:

1. maxSkillProbes truncating before Exclude - caps now apply after exclusion (probeFamilyCap, expressed as probe counts). Test covers accept-then-regenerate reaching the skills past the cap.
2. Approval probe on restricted - it is supervised-only now. Restricted has no use_tools permission, so the engine blocks before policy is consulted.
3. Mid-batch failure leaving saved probes acceptable - cards are hidden as each write lands. Same bug existed in SuggestCases.svelte, fixed there too.
4. cancelNewSet retargeting without reloading - it re-runs the pass now, since exclusion is server-side and keyed on the set.
5. Near-miss skill probe grading an impossible failure - notes now grade the explanation against the skill's own frontmatter.
6. Agent-less instances - panel says to pick an agent rather than firing a guaranteed 400, and both CTAs are disabled until there is one.
7. Duplication with SuggestCases.svelte - deliberately left out of this PR. Both copies of the batch bug are fixed, but extracting the shared target-set/accept machinery is a refactor across two live surfaces and I would rather do it as its own change than bury it under a probe feature. Happy to do it next if you want it before merge.

On cadence's flag: probes quote configuration back (the tier, a skill's description, tool names, the auto-approve list), so eval:read alone was a way around the scopes that guard reading it. GET /eval/probes now needs agents:read as well (403 without), and drops skill_instruction without skills:read and approval_policy without tools:read via ProbeOpts.SkipKinds.

just hook and just test-integration pass. New tests: family cap after exclusion, no approval probe on restricted, scope requirement, family narrowing by scope, and three Vitest cases for the UI fixes.

[Temikus]

Follow-up in 616274c on top of fe410bf, after re-verifying every finding against the branch: the near-miss probe's notes still asked for an account of the skill "from its written instructions", which an unmatched-skill turn cannot honestly give (commandMatches fires on the first token only, so the skill body is never injected). The notes now grade what is actually observable — treating the mention as an ordinary question and not claiming to have run the skill — with the Go tests updated. Also added the vitest coverage the SuggestCases batch fix landed without (saved case hidden after a mid-batch failure, re-accept retries only what never landed, no double POST).

Everything else in fe410bf checked out as described; just hook green on the pushed head.

---

Generated by Claude Code