feat(eval): add auditable native matrix runner

[vincentkoc]

What does this PR do?

Adds a native ShellBench matrix runner with Harbor-compatible task execution, results, manifests, checkpoints, and aggregation.

Why?

The July 27 native matrix exposed three ranking-integrity gaps: planned provider identity could drift at dispatch, non-Codex routes emitted synthetic trajectories, and duplicate or unexpected task results could enter scoring. Artifact pulls also needed crash-safe recovery.

Fixes #41.

Changes

• add AWS Crabbox fleet planning, dispatch, checkpointing, final export, and resume support
• preserve planned and observed model identity through dispatch and result manifests
• record validated real Codex events; keep unsupported harness routes explicitly unranked
• require exact expected-task reconciliation and exclude duplicate or unexpected results from scoring
• make checkpoint and final downloads atomic and recover completed exports after controller interruption
• add audit, proxy, aggregation, runtime, fleet, and checkpoint tests

Tests

• python -m pytest -q passes locally
• python -m ruff check clawbench app.py scripts tests passes locally, or the change is docs-only

Focused validation: 55 passed; Ruff, bash -n, shellcheck, and git diff --check passed. ShellCheck reports only the existing informational /etc/os-release source notice.

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed July 27, 2026, 1:34 PM ET / 17:34 UTC.

ClawSweeper review

What this changes

Adds a native ShellBench matrix runner that plans and executes multi-harness benchmark jobs, records provenance and real Codex trajectories, checkpoints/export results, and aggregates only reconciled eligible results.

Merge readiness

⚠️ Ready for maintainer review - 5 items remain

Keep this draft PR open. It is a substantial new evaluation and fleet-control capability linked to the still-open canonical request, and adopting its privileged AWS/Crabbox execution path and leaderboard-integrity contract needs explicit product and operational approval rather than cleanup closure.

Priority: P3
Reviewed head: 682b6c7847b4d96dd39fdb2e0092dd90d97453f3
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🦐 gold shrimp (3/6)	The branch shows substantial focused test coverage and CI success, but acceptance confidence is limited by the new privileged execution surface and absence of reviewed end-to-end fleet evidence.
Proof confidence	🌊 off-meta tidepool	Not applicable: This member-authored draft is exempt from the external-contributor proof gate; reported unit and static checks are supplemental, while the remaining acceptance need is maintainer-reviewed end-to-end fleet evidence.
Patch quality	🦐 gold shrimp (3/6)	Security review found an item that needs attention.

Verification

Check	Result	Evidence
Real behavior	Not applicable	Not applicable: This member-authored draft is exempt from the external-contributor proof gate; reported unit and static checks are supplemental, while the remaining acceptance need is maintainer-reviewed end-to-end fleet evidence.
Evidence reviewed	5 items	PR scope: The branch adds 21 files and 7,867 lines, including a new native evaluator, remote fleet controller, bootstrap scripts, checkpoint/export handling, and six focused test modules; this is a new operational subsystem rather than a narrow repair. Canonical work remains open: The PR explicitly targets the open native-matrix request at #41; that issue describes the same fleet scheduling, provenance, checkpointing, aggregation, and Harbor-parity goals. Integrity boundary: The aggregation and runtime paths add explicit eligibility, expected-task reconciliation, and trajectory validation, so their acceptance determines whether native outputs may influence benchmark rankings.
Findings	None	None.
Security	Needs attention	Review remote bootstrap trust boundary: The new bootstrap script installs Docker and evaluation toolchains as root on remote workers, so maintainers should verify artifact provenance, credential handling, and the intended isolation boundary before operating it with production provider keys.

How this fits together

ShellBench evaluation jobs turn a task suite and planned model/harness matrix into remote benchmark runs, archived artifacts, and aggregated ranking data. This PR introduces a native AWS/Crabbox execution route alongside Harbor-compatible artifacts and eligibility checks.

flowchart LR
  A[Task suite and matrix plan] --> B[Native fleet controller]
  B --> C[AWS Crabbox workers]
  C --> D[Harness and proxy execution]
  D --> E[Run manifests and checkpoints]
  E --> F[Result reconciliation]
  F --> G[Eligible aggregate reports]

Loading

Decision needed

Question	Recommendation
Should ShellBench support this AWS/Crabbox native matrix runner as an official evaluation route whose outputs may be calibrated for leaderboard use alongside Harbor?	Stage it behind an experimental workflow: Approve a bounded native-run experiment with documented non-ranking defaults until an end-to-end parity package demonstrates eligibility.

Why: The patch is an implementation of a new operational and ranking-integrity contract, not a pre-existing narrowly defined behavior; maintainers must choose the supported execution and calibration boundary before code review alone can settle acceptance.

Before merge

• Resolve security concern: Review remote bootstrap trust boundary - The new bootstrap script installs Docker and evaluation toolchains as root on remote workers, so maintainers should verify artifact provenance, credential handling, and the intended isolation boundary before operating it with production provider keys.
• Resolve merge risk (P1) - Merging would introduce a privileged remote-worker provisioning and Docker-installation path; its expected execution environment, credential exposure model, and third-party toolchain trust boundary need explicit operational approval.
• Resolve merge risk (P1) - The new eligibility and reconciliation rules can affect which native benchmark outputs are ranked, so maintainers need an agreed Harbor-parity and audit policy before treating results as comparable.
• Resolve merge risk (P1) - The branch is a draft and adds a large new subsystem; green unit and static checks do not establish an end-to-end native run, checkpoint recovery, and aggregate result from an actual fleet.

Findings

• [medium] Review remote bootstrap trust boundary — scripts/native_eval/bootstrap_beast.sh:1

Agent review details

Security

Needs attention: The patch adds a root-level remote bootstrap and third-party toolchain installation path, which warrants operational security review before the workflow is adopted.

Review metrics

Metric	Value	Why it matters
New execution surface	21 files added; 7,867 additions; 0 deletions	The PR introduces a complete runner and remote-control subsystem rather than modifying an existing small execution path.
Focused validation	6 new native-evaluation test modules; 23 commits	The branch has substantial incremental test coverage, but its size and operational scope still warrant end-to-end acceptance evidence.

Merge-risk options

Maintainer options:

1. Require an experimental acceptance boundary (recommended)
   Before merge, document that native output is unranked by default and provide a redacted end-to-end calibration and recovery proof for the proposed supported path.
2. Accept operational ownership now
   Merge with an explicit maintainer decision that the repository owns the worker bootstrap, credentials, artifact retention, and ranking-calibration policy.
3. Pause the core integration
   Keep the implementation out of the default repository workflow if maintainers do not want to own an AWS/Crabbox evaluation service.

Technical review

Best possible solution:

Adopt the native route only after maintainers define its supported operating boundary and ranking policy, then land it with a redacted end-to-end Crabbox run showing manifest provenance, recovery from a checkpoint, and an eligible aggregate export.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this PR proposes a new native evaluation route rather than reporting a current broken behavior; the relevant validation is an end-to-end fleet and artifact-recovery demonstration.

Is this the best way to solve the issue?

Unclear: the implementation is coherent with the linked proposal, but adding a repository-owned AWS/Crabbox runner is not yet shown to be the best supported product boundary compared with an experimental or externally operated route.

AGENTS.md: unclear because the file could not be read completely.

Codex review notes: model internal, reasoning high; reviewed against 28ce18e5dfc8.

Labels

Label changes:

• add P3: This is a speculative new evaluation capability and operational workflow, not a current user-facing outage or regression.
• add merge-risk: 🚨 other: Native result eligibility and reconciliation can change benchmark-ranking trust and operational ownership in ways ordinary CI does not resolve.
• add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🌊 off-meta tidepool and patch quality is 🦐 gold shrimp.
• add status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Not applicable: This member-authored draft is exempt from the external-contributor proof gate; reported unit and static checks are supplemental, while the remaining acceptance need is maintainer-reviewed end-to-end fleet evidence.

Label justifications:

• P3: This is a speculative new evaluation capability and operational workflow, not a current user-facing outage or regression.
• merge-risk: 🚨 other: Native result eligibility and reconciliation can change benchmark-ranking trust and operational ownership in ways ordinary CI does not resolve.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🌊 off-meta tidepool and patch quality is 🦐 gold shrimp.
• status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Not applicable: This member-authored draft is exempt from the external-contributor proof gate; reported unit and static checks are supplemental, while the remaining acceptance need is maintainer-reviewed end-to-end fleet evidence.

Evidence

Security concerns:

• [medium] Review remote bootstrap trust boundary — scripts/native_eval/bootstrap_beast.sh:1
  The new bootstrap script installs Docker and evaluation toolchains as root on remote workers, so maintainers should verify artifact provenance, credential handling, and the intended isolation boundary before operating it with production provider keys.
  Confidence: 0.88

What I checked:

• PR scope: The branch adds 21 files and 7,867 lines, including a new native evaluator, remote fleet controller, bootstrap scripts, checkpoint/export handling, and six focused test modules; this is a new operational subsystem rather than a narrow repair. (scripts/native_eval/fleet.py:1, 682b6c7847b4)
• Canonical work remains open: The PR explicitly targets the open native-matrix request at Add an auditable native ShellBench matrix runner #41; that issue describes the same fleet scheduling, provenance, checkpointing, aggregation, and Harbor-parity goals. (scripts/native_eval/aggregate.py:1, 682b6c7847b4)
• Integrity boundary: The aggregation and runtime paths add explicit eligibility, expected-task reconciliation, and trajectory validation, so their acceptance determines whether native outputs may influence benchmark rankings. (scripts/native_eval/aggregate.py:1, 682b6c7847b4)
• Operational security surface: The worker bootstrap script runs with root privileges and installs/configures Docker plus external toolchains on remote workers; this needs focused review of the intended trust and isolation model before use. (scripts/native_eval/bootstrap_beast.sh:1, 682b6c7847b4)
• Validation reported by contributor: The PR reports 55 focused tests plus Ruff, shell syntax, ShellCheck, and diff checks; GitHub context also reports successful Python, CodeQL, dispatch, and analysis checks. This is useful supplemental validation, but not a decision on whether the native route should become a supported product surface. (tests/test_native_eval_fleet.py:1, 682b6c7847b4)

Likely related people:

• vincentkoc: Authored the linked native-matrix proposal and the PR's 23 incremental commits covering runner, fleet, provenance, checkpoint, aggregation, and result-validity behavior. (role: native evaluation area contributor; confidence: medium; commits: 97932cf8f864, 164427a3722c, d2c31efbd6bc; files: scripts/native_eval/fleet.py, scripts/native_eval/runtime.py, scripts/native_eval/aggregate.py)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Provide a redacted live native run showing planned versus observed model identity, checkpoint recovery, and final eligibility output.
• Document whether native results are unranked by default until a Harbor parity calibration is approved.
• Have a maintainer explicitly approve or narrow the remote-worker provisioning boundary.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.