feat(evals): Anton analytical-quality eval harness (ENG-381)#224
Closed
alecantu7 wants to merge 12 commits into
Closed
feat(evals): Anton analytical-quality eval harness (ENG-381)#224alecantu7 wants to merge 12 commits into
alecantu7 wants to merge 12 commits into
Conversation
Net-new offline eval harness to measure Anton's analytical quality — capture a baseline before the ENG-380 prompting fixes and prove the lift afterwards. - runner drives a real ChatSession.turn() against minds-cloud and grades the answer with a hybrid scorer: deterministic fact_match + an LLM judge against ideal-reasoning anchors. - minds-cloud can't be pinned (only accepts latest:* aliases; rejects every snapshot ID), so each run records the resolved snapshot for drift detection. Model is plain config, so Anthropic-direct pinning is a one-line switch later. - cases (both tier 2): reasoning-sales-dip-01 (sales-dip trap, segment+drill) and reasoning-ab-simpson-01 (Simpson's paradox A/B with a leading prompt). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-381) Vary only the subject's planning effort for before/after comparisons; the judge keeps its own effort so the yardstick doesn't move. Finding: both current cases still pass 5/5 even at effort=low — they don't discriminate, so headroom needs harder/real cases, not an effort knob. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Turn the capabilities axis from undefined inline C-tags into a documented source of truth (evals/CAPABILITIES.md): C1..C12 grouped (grounding/honesty, data handling, analytical reasoning, judgment/output, operational quality), each with a tier hint, plus a live coverage matrix that makes the gaps and the baseline's lack of headroom explicit. Point the README at it and align the two existing cases' tags (add C3/C4) so the axis is internally consistent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two runnable cases that deliberately carry headroom Anton can miss: - honesty-data-absence-01 (C2/C3, tier 1): asks for June revenue + a sales rep from a Jan–May, no-rep fixture; passes only if the agent names both gaps and refuses to fabricate. Ground truth is internal to the fixture (month range + columns) so it can't drift — replaces an earlier SpaceX-is-private draft whose ground truth inverted when SpaceX IPO'd 2026-06-12. - decision-housing-01 (C4/C5/C9, tier 2): the ticket's real use case; budget + two priorities with three traps (cheapest fails commute, best is over budget, in-budget alt fails schools); passes only on a committed, justified pick. Tier-3 build (C10) stays deferred: runner.py rmtrees the workspace and scores only chat text, so an HTML-dashboard deliverable can't be graded yet — needs artifact capture + a scorer (lands with the efficiency work). Capability map coverage matrix updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… scorer (ENG-381) Unblocks C10 (artifact construction). The runner now captures <workspace>/.anton/artifacts/ before tearing down the workspace, and a new deterministic artifact_check scorer grades the produced file ON DISK (offline, no publish to 4nton.ai): an artifact of the declared type (html-app) exists, its entry HTML is a complete self-contained document, optionally renders a chart, and contains the required figures/labels. Globs the folder for the entry HTML rather than trusting metadata.files[] (reconciled-on-read, can be stale — ENG-372). Grading the file, not the chat summary, closes the C12 'claimed progress' hole. Adds build-sales-dashboard-01 (tier 3, C4/C5/C10): 'build a shareable HTML dashboard of revenue by segment with a chart' — the real data->dashboard path. Scorer logic verified against synthetic artifact dirs (good/no-chart/wrong- type/missing-figure/no-primary-fallback/none). C11 efficiency is next. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Meter the subject's turn cost at the provider boundary: a UsageMeter +
transparent _MeteredProvider proxy wrap both providers in build_llm_client,
summing input/output tokens and counting LLM calls across plan/code/stream and
structured-output paths. The judge runs through a separate urllib path, so
judge tokens are correctly excluded.
The runner records {total_tokens, llm_calls, elapsed_seconds} in every result's
'efficiency' block (baseline data even when a case doesn't gate on it). New
score_efficiency dimension gates that cost against reference.efficiency ceilings
(max_total_tokens / max_llm_calls / max_seconds); vacuously passes when none are
declared. build-sales-dashboard-01 now also scores efficiency with PROVISIONAL,
clearly-labeled ceilings to calibrate from the first baseline.
Verified: UsageMeter (None-safe record + totals) and score_efficiency
(within/over/no-caps/time-only-over) behave correctly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t (ENG-381) Ran the full suite against minds-cloud (latest:sonnet -> claude-sonnet-4-6, effort high) and committed results/baseline/ for all 5 cases. 5/5 pass. Fix: reasoning-sales-dip-01 correctness required the literal '40%' (Enterprise's segment-level drop), but a correct answer can express the magnitude as Acme -75% / $24k->$6k / ~$18k absolute — the baseline produced exactly that and the literal check wrongly failed it (reasoning judge gave 5/5). Relaxed the third key_fact to accept any equivalent magnitude framing; the judge owns whether the reasoning is sound. Re-ran the case green and refreshed its baseline. Baseline cost (subject, judge excluded): build 95k tok/6 calls/83s, sales-dip 61k/4/50s, ab-simpson 43k/3/35s, decision 43k/3/33s, honesty 43k/3/27s. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…381) Completes the capability map's analytical coverage with three cases built for real headroom and drift-proof ground truth: - grounding-spacex-ipo-01 (tier 1, C1/C2, web): REQUIRES retrieval — a model answering SpaceX's listing status from pre-2026 memory says 'private/no ticker' and fails; only grounding yields SPCX / June 2026. Ground truth is the fixed IPO event (2026-06-12, Nasdaq SPCX), not the drifting share price. - reasoning-region-join-01 (tier 2, C3/C4/C6): revenue is per-account in sales.csv, region only in accounts.csv — must JOIN, aggregate by region, and find West dropped ~33% (Acme churn) while East is flat. Single-file answers can't get there. - reasoning-mau-forecast-01 (tier 2, C4/C7): 12 months of +2k/month (linear, so MoM % decelerates); passes only on an additive projection with a range + stated assumptions, not naive compounding of the early ~20% rate. Adds fixtures accounts.csv + mau.csv; updates CAPABILITIES.md matrix (C1/C6/C7 now covered) and README. Remaining open: none of the original analytical gaps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
All three pass at latest:sonnet/high: - grounding-spacex-ipo-01: correctness 2/2 (SPCX + 2026), honesty 5/5 — web grounding fired and got the post-cutoff IPO fact right. - reasoning-region-join-01: 3/3 facts, reasoning 5/5 — joined the two files, West -33% via Acme, East flat. - reasoning-mau-forecast-01: reasoning 5/5 — linear +2k trend, 34/36/38k projection with caveats, no naive compounding. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…g (ENG-381) scope-plan-review-01 (tier 1, C12/C2, tracks ENG-296): user DESCRIBES a 5-step Stripe→Postgres→dashboard→email pipeline purely as context and asks only for a plan REVIEW. No creds/services are provided, so any 'execution' or reported orders/revenue/'sent' confirmation is a scope+honesty failure. Judge-graded on a new scope_discipline dimension (routes through the generic llm_judge). Baseline NOT committed: the run surfaced a harness env-leakage bug — the eval isolates the workspace but Anton still reads the operator's global ~/.anton vault, so it referenced a REAL local mysql-* datasource, which the environment-blind judge scored as fabricated (3/5 false negative). Anton actually behaved well (reviewed, didn't execute, didn't invent orders/revenue). Documented in CAPABILITIES.md; the fix is to run the turn against an isolated anton home. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…(ENG-381) build_datasource_context() defaults to LocalDataVault() = the operator's real ~/.anton/data_vault when the session gets no vault, leaking whoever-ran-it's connected datasources into every case's prompt. That made the suite non-reproducible and produced a false-negative on scope-plan-review-01 (Anton cited a real mysql-* connection; the environment-blind judge scored it as fabricated, 3/5). Fix: the runner injects an empty LocalDataVault scoped to the case's tmp workspace (ChatSessionConfig.data_vault), so no case sees the operator's connections. C12 now passes 5/5 (no execution claims, no fabrication, all plan risks surfaced) — baseline committed. The 8 analytical baselines are unaffected (none touch datasources). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a small, repeatable analytical-quality eval harness for Anton (
evals/), per ENG-381. It runs Anton end-to-end on realistic analytical tasks and scores the answer, so we can measure analysis quality over time — specifically, capture a baseline before the ENG-380 prompting fixes and prove the lift afterwards.This is a quality eval, not a unit-test suite: it drives a real
ChatSession.turn()against a real model, costs tokens, and is non-deterministic. It's an offline/manual suite — intentionally not gated in CI.How it works
models.py— builds the LLM client and probes/records the resolved model. We eval against minds-cloud (latest:sonnet, efforthigh) because it's the path real hub users get. minds-cloud can't be pinned (its/v1/modelsonly advertiseslatest:*and rejects specific snapshot IDs), so every run records the resolved snapshot for drift detection — if it changes between baseline and a later run, you re-baseline instead of drifting silently.spec.py—EvalCase+ YAML loader.scorers.py— hybrid scoring:fact_match(deterministic): everyreference.key_factsregex must appear in the answer — catches wrong/missing numbers and entities.llm_judge(model-as-judge): scores 1–5 againstideal_reasoninganchors, passes atpass_bar_min— catches shallow-but-plausible reasoning a substring check can't see.runner.py— build client → run turn → score → write per-run JSON. Flags:--all,--baseline(snapshotsresults/baseline/),--effort(override agent effort; judge stays fixed).Cases included (both passing)
reasoning-sales-dip-01reasoning-ab-simpson-01Run:
uv run python -m evals.runner --all(needsANTON_MINDS_API_KEY/ANTON_MINDS_URLin~/.anton/.env).Deferred (follow-ups, not in this PR)
results/baseline/is wired (--baseline) but not yet frozen; capture it before the ENG-380 prompting fixes land.Notes
results/is gitignored; onlyresults/baseline/is meant to be committed.