fix(eval): preserve traces and reasoning controls

[vincentkoc]

What does this PR do?

Preserves complete native OpenClaw, Hermes, and Codex traces, converts them to Harbor-compatible ATIF-v1.7 trajectories, and makes agent, judge, repair, and post-processing provenance independently auditable.

Why?

The first GPT-5.6 OpenClaw/Hermes matrix completed all tasks, but native trajectory export was incomplete. A later behavioral audit found that baseline BOOTSTRAP.md setup could push OpenClaw into onboarding instead of task execution. Codex also exposed a trace collector edge case when binary tool output corrupted the CLI JSONL stream even though the native session rollout remained complete.

The final audit then found two post-processing defects: verifier scorecard.json infra statuses were not included in repair planning, and aggregate task names trusted display titles instead of canonical task_id.path leaves.

Changes

• archive complete isolated OpenClaw sessions and recover canonical transcripts
• export Hermes sessions when the CLI omits a session id and parse JSONL on LF boundaries
• recover active AWS leases after controller restarts
• convert native events to Harbor-compatible ATIF-v1.7
• preserve canonical provider model identity
• accept OpenAI xhigh reasoning effort while keeping the judge pinned independently at high
• set OpenClaw skipBootstrap, remove BOOTSTRAP.md, and detect markerless final envelopes
• recover Codex trajectories from native session rollouts when CLI JSONL is malformed or binary-corrupted
• replace degraded stored leases without discarding recovered artifacts
• canonicalize aggregate task identities from task_id.path

Validation

• PYTHONPATH=. pytest -q tests/test_native_eval*.py (395 passed, 5 skipped on the full runner series)
• focused aggregate/fleet/checkpoint suite on final head (53 passed)
• focused campaign repair/publication suite (24 passed)
• focused Ruff checks on changed runner and campaign modules
• signed commits verified locally
• GitHub Python 3.11 and 3.12 CI passes on f2633f26c35958b3aacd15702712a60318ad624b
• live AWS OpenClaw probe reached the real task prompt with bootstrap absent and canonical identity preserved
• low/medium/xhigh agent routes remain separate while shellbench-judge stays at high
• final targeted repair campaign: 79 successful repair runs retained alongside one broken plan and one lease-lost attempt
• 66 canonical runs received exact-task overlays; all source and superseded artifacts remain preserved
• final campaign audit: 93/93 canonical runs, 10,788/10,788 task trials, zero infra rows, zero missing keys, zero unexpected keys, and zero unresolved canonical slots
• all 93 selected final archives pass SHA-256, tar integrity, and exact 116-result checks
• publication secret scans report zero exact-value findings and zero verified TruffleHog credentials

Result publications

• https://github.com/ShellBench/public-tasks/pull/40
• https://github.com/ShellBench/public-tasks/pull/43
• https://github.com/ShellBench/public-tasks/pull/44
• https://github.com/ShellBench/public-tasks/pull/45

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed July 28, 2026, 11:16 AM ET / 15:16 UTC.

ClawSweeper review

What this changes

This PR expands ShellBench’s native evaluation runner to export OpenClaw and Hermes trajectories, independently route judge reasoning, avoid OpenClaw bootstrap contamination, and recover/finish remote runs more reliably.

Merge readiness

⚠️ Ready for maintainer review - 5 items remain

Keep this draft open for maintainer review rather than cleanup closure: it is a member-authored, stacked native-evaluation change with a viable implementation direction but outstanding integrated validation. Likely related people: vincentkoc is the strongest routing candidate from the current native-evaluation commit series and prerequisite stack.

Priority: P2
Reviewed head: 2a71ebd40fd12770c175b00614c1448a3caac1b3
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🦐 gold shrimp (3/6)	The branch has substantial focused implementation and reported runtime evidence, but its stacked scope and incomplete integrated matrix audit limit merge confidence.
Proof confidence	🌊 off-meta tidepool	Not applicable: The author is a repository MEMBER, so the external-contributor proof gate does not apply; nevertheless, the PR body reports live AWS, proxy, and lifecycle probes, while the completed full matrix audit remains the key merge-validation gap.
Patch quality	🦐 gold shrimp (3/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Not applicable	Not applicable: The author is a repository MEMBER, so the external-contributor proof gate does not apply; nevertheless, the PR body reports live AWS, proxy, and lifecycle probes, while the completed full matrix audit remains the key merge-validation gap.
Evidence reviewed	5 items	Current PR scope: The supplied PR metadata identifies 15 changed files with 1,714 additions and 60 deletions, spanning the native runner, fleet recovery, proxy routing, trace conversion, remote archive scripts, and focused tests. Focused behavioral coverage: The branch adds focused tests for xhigh effort, judge-route isolation, markerless OpenClaw completion, lease recovery, trajectory eligibility, and trace handling; the PR body also reports passing Python CI and targeted native-evaluation tests. Stack dependency: This PR is explicitly stacked on the still-open prerequisite #44, and its base ref is that prerequisite branch rather than current main. The remaining review gate is therefore the combined post-stack behavior, not an obsolete branch cleanup.
Findings	None	None.
Security	None	None.

How this fits together

ShellBench’s native-evaluation pipeline plans model runs, starts remote harnesses through the fleet controller, gathers native session output, converts it into comparable trajectories, and aggregates scored results. This PR changes the runner and proxy layers that determine whether benchmark outputs are complete, correctly attributed, and safe to compare.

flowchart LR
  A[Evaluation plan] --> B[Proxy model routes]
  B --> C[Remote fleet controller]
  C --> D[OpenClaw or Hermes harness]
  D --> E[Native logs and sessions]
  E --> F[Trajectory conversion]
  F --> G[Aggregation and leaderboard]
  C --> H[Checkpoint and final archives]

Loading

Decision needed

Question	Recommendation
Should this broad native-evaluation stack land as one post-#44 change after a completed integrated matrix audit, or should its independent fixes be split before merge?	Rebase and validate the complete stack: Land the prerequisite, rebase this PR, and require one completed redacted end-to-end matrix/lifecycle audit before merging the combined behavior.

Why: The correct merge boundary determines benchmark comparability and remote-run operational risk; isolated unit coverage does not settle the behavior of the stacked fleet, proxy, harness, archive, and aggregation path.

Before merge

• Resolve merge risk (P1) - The branch is stacked on the still-open prerequisite feat(eval): add GPT-5.6 Luna and Terra models #44, and the corrected 93-job combined behavioral and artifact audit remains unchecked in the PR body.
• Resolve merge risk (P1) - Trajectory eligibility and dedicated judge routing change how existing native runs are considered comparable and complete; an upgrade or rerun must not silently mix results produced under different semantics.
• Resolve merge risk (P1) - Markerless completion detection and active-lease recovery affect remote-agent lifetime and cleanup, so the full stack needs a real lifecycle run before merge.
• Complete next step (P2) - Maintainers need to choose the stack boundary and require the completed integrated runtime audit before merging; this is not a narrow automated repair task.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Patch surface	15 files; 1,714 added, 60 removed	The change crosses evaluation planning, remote execution, artifact capture, trace conversion, and aggregation rather than a single isolated harness fix.
Commit stack	12 commits on an open prerequisite branch	Merge order and integrated validation matter because this PR depends on #44.
Reported live checks	3 live probes plus focused test and CI claims	The PR reports real OpenClaw, proxy, and AWS lifecycle observations, but the final full matrix audit remains incomplete.

Merge-risk options

Maintainer options:

1. Validate the rebased benchmark stack (recommended)
   After the prerequisite lands, rebase this branch and attach a completed real matrix/lifecycle audit showing comparable trajectories, fixed judge effort, and clean completion behavior.
2. Narrow the landing surface
   Pause the combined branch and split changes whose runtime evidence can be reviewed independently if the integrated matrix remains impractical to complete.

Technical review

Best possible solution:

Merge the prerequisite stack in order, rebase this branch onto its landed base, then attach a redacted completed-matrix and lifecycle artifact audit that proves trace completeness, fixed judge effort, clean agent termination, and preserved archive access together.

Do we have a high-confidence way to reproduce the issue?

Not applicable as an issue reproduction: this PR reports live AWS, proxy, and lifecycle probes for several changed paths, but its corrected full matrix audit is still pending.

Is this the best way to solve the issue?

Unclear: the individual repairs and focused coverage are coherent, but the broad stacked implementation is only the best landing path once maintainers accept the combined benchmark semantics and integrated operational proof.

AGENTS.md: unclear because the file could not be read completely.

Codex review notes: model internal, reasoning high; reviewed against 28ce18e5dfc8.

Labels

Label changes:

• add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🌊 off-meta tidepool and patch quality is 🦐 gold shrimp.
• remove rating: 🐚 platinum hermit: Current PR rating is rating: 🦐 gold shrimp, so this older rating label is no longer current.

Label justifications:

• P2: This is a substantial benchmark-runner improvement with bounded operational blast radius, but it is not an established production outage or security incident.
• merge-risk: 🚨 compatibility: The patch changes trajectory-completeness and canonical-model eligibility plus judge routing, which can change how native benchmark results compare with existing runs.
• merge-risk: 🚨 availability: The patch changes remote completion detection and active-lease recovery, so an error can leave agents running or end a wrapper at the wrong time.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🌊 off-meta tidepool and patch quality is 🦐 gold shrimp.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The author is a repository MEMBER, so the external-contributor proof gate does not apply; nevertheless, the PR body reports live AWS, proxy, and lifecycle probes, while the completed full matrix audit remains the key merge-validation gap.

Evidence

What I checked:

• Current PR scope: The supplied PR metadata identifies 15 changed files with 1,714 additions and 60 deletions, spanning the native runner, fleet recovery, proxy routing, trace conversion, remote archive scripts, and focused tests. (scripts/native_eval/runtime.py:1, 2a71ebd40fd1)
• Focused behavioral coverage: The branch adds focused tests for xhigh effort, judge-route isolation, markerless OpenClaw completion, lease recovery, trajectory eligibility, and trace handling; the PR body also reports passing Python CI and targeted native-evaluation tests. (tests/test_native_eval_runner.py:1, 2a71ebd40fd1)
• Stack dependency: This PR is explicitly stacked on the still-open prerequisite feat(eval): add GPT-5.6 Luna and Terra models #44, and its base ref is that prerequisite branch rather than current main. The remaining review gate is therefore the combined post-stack behavior, not an obsolete branch cleanup. (scripts/native_eval/models.py:99, 96b6c9363d76)
• Recent feature provenance: The current head follows a 12-commit native-evaluation series, including dedicated commits for judge isolation, bootstrap avoidance, markerless completion, and trajectory eligibility. The behavior appears to be actively developed rather than superseded. (scripts/native_eval/harnesses.py:68, 2a71ebd40fd1)
• Repository-policy inspection limitation: The available read-only command sandbox failed before repository files could be opened, so the full repository AGENTS.md policy and current-main source could not be independently verified. This supports a conservative keep-open decision and prevents any implementation-on-main conclusion.

Likely related people:

• vincentkoc: Authored the active 12-commit series across the runner, fleet, proxy, trajectory, and regression-test surfaces, and also authored the open prerequisite stack branch. (role: native evaluation contributor; confidence: medium; commits: 6737e858a0ab, 242c5d6d85d8, d676d3c28891; files: scripts/native_eval/runtime.py, scripts/native_eval/harnesses.py, scripts/native_eval/proxy.py)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Land or otherwise resolve feat(eval): add GPT-5.6 Luna and Terra models #44, then rebase this branch onto the resulting base.
• Attach a redacted completed matrix and lifecycle audit that covers the combined proxy, harness, trajectory, archive, and fleet paths.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (10 earlier review cycles; latest 8 shown)

• reviewed 2026-07-28T06:40:26.951Z sha 8c32eac :: needs maintainer review before merge. :: none
• reviewed 2026-07-28T07:41:26.627Z sha c6f246f :: needs maintainer review before merge. :: none
• reviewed 2026-07-28T09:21:28.204Z sha 1e1d042 :: needs maintainer review before merge. :: none
• reviewed 2026-07-28T11:02:42.886Z sha 23bc37f :: needs maintainer review before merge. :: none
• reviewed 2026-07-28T11:17:21.866Z sha 242c5d6 :: needs maintainer review before merge. :: none
• reviewed 2026-07-28T11:29:24.190Z sha 7b0fc61 :: needs maintainer review before merge. :: none
• reviewed 2026-07-28T11:37:36.581Z sha b3d340c :: found issues before merge. :: [P3] Remove release-owned changelog edits
• reviewed 2026-07-28T11:54:29.581Z sha b3d340c :: needs maintainer review before merge. :: none

[vincentkoc]

Superseded by #46, which replayed and landed this branch’s still-unmerged commits after the parent squash made the protected stack impossible to update in place.