fix(native-eval): benchmark genuine OpenClaw code mode

[vincentkoc]

What does this PR do?

Makes the OpenClaw native-eval arms represent genuine direct, directory, and
Code Mode execution, then exports and validates their public trajectory bundles
before a run can be scored.

Why?

Fixes #61.

The earlier code arm selected the legacy tool_search_code bridge rather than
OpenClaw Code Mode. The harness also masked setup failures, forced thinking off,
terminated the runtime before lifecycle cleanup settled, and reconstructed
delegated traces from private session files. Those failures made the released
direct/code comparison invalid and disproportionately erased Code Mode's nested
tool calls.

Changes

• Map direct, directory, and code to explicit, mutually exclusive
  OpenClaw tool surfaces.
• Propagate the requested reasoning effort through planning, dispatch,
  OpenClaw defaults, subagents, CLI execution, and manifests.
• Fail setup immediately and let openclaw agent exit naturally.
• Export root and child trajectories through
  openclaw sessions export-trajectory.
• Capture child runs at the canonical pre-cleanup terminal hook, including
  nested and repeated runs, without blocking the Gateway.
• Validate audit initialization, trace/session identity, event counts, terminal
  status, complete Code Mode snapshots, and the exact provider-visible
  exec/wait surface.
• Reconstruct Code Mode nested calls and complete session-tree usage from the
  public export bundles.
• Bound child-export stabilization and reject missing or failed evidence rather
  than publishing partial traces.

Tests

• Blacksmith Testbox full suite: 454 passed, 5 skipped
• Blacksmith Testbox focused runner suite after final fixes: 61 passed
• Ruff clean
• Python compile, git diff --check, generated shell syntax for all three
  modes, and generated audit-plugin node --check
• Fresh Codex autoreview: clean
• Fresh matched direct/Code Mode ShellBench batch

The July 29, 2026 released code result remains legacy Tool Search bridge
evidence only. It is not evidence about genuine OpenClaw Code Mode.

[clawsweeper]

Codex review: found issues before merge. Reviewed August 2, 2026, 9:16 AM ET / 13:16 UTC.

ClawSweeper review

What this changes

The PR changes ShellBench’s OpenClaw native-evaluation runner to select explicit direct, directory, and Code Mode tool surfaces, export public root and child trajectories, and validate those exports before scoring benchmark runs.

Merge readiness

⛔ Blocked by patch quality or review findings - 6 items remain

This member-authored PR should remain open. It still has the previously reported P1 compatibility regression: established native-evaluation configuration keys become hard failures without a tested translation or approved migration path.

Priority: P2
Reviewed head: f5b31cb1260b631b811bf0ede9f4eba51a94b98c

Review scores

Measure	Result	What it means
Overall readiness	🦪 silver shellfish (2/6)	The branch has substantial focused tests, but the unresolved P1 upgrade regression makes it unready to merge.
Proof confidence	🌊 off-meta tidepool	Not applicable: This member-authored PR is not subject to the external-contributor proof gate; however, the unchecked matched benchmark batch remains useful release-confidence evidence.
Patch quality	🦪 silver shellfish (2/6)	1 actionable review finding remain.

Verification

Check	Result	Evidence
Real behavior	Not applicable	Not applicable: This member-authored PR is not subject to the external-contributor proof gate; however, the unchecked matched benchmark batch remains useful release-confidence evidence.
Evidence reviewed	5 items	Legacy environment input now fails: The branch raises for any non-empty legacy environment key instead of translating supported values to the new explicit tool modes, so existing jobs can stop before execution. Legacy fleet-plan input now fails: The branch also rejects the legacy fleet manifest key and tells operators to replace it, but does not provide conversion or upgrade compatibility. Prior finding remains at the same head: The latest completed review identified this upgrade-path blocker at the same reviewed head; comparing that head to itself confirms no intervening change to the affected paths.
Findings	1 actionable finding	[P1] Preserve an upgrade path for retired tool-mode configuration
Security	None	None.

How this fits together

ShellBench turns fleet plans and environment settings into OpenClaw benchmark jobs, then validates exported session trajectories before publishing scores. The changed native-evaluation path bridges run configuration, OpenClaw execution, trajectory evidence, and benchmark results.

flowchart LR
  A[Fleet plans and environment] --> B[Native evaluation runner]
  B --> C[OpenClaw tool configuration]
  C --> D[Benchmark agent sessions]
  D --> E[Public trajectory exports]
  E --> F[Trajectory validation]
  F --> G[Scored benchmark results]

Loading

Before merge

• Preserve an upgrade path for retired tool-mode configuration (P1) - SHELLBENCH_OPENCLAW_TOOL_SEARCH_MODE now raises before a run starts, and the parallel fleet-plan key is rejected too. Existing native-evaluation automation can therefore stop on upgrade rather than receive the intended explicit mode. This concern was visible at the prior reviewed head and remains unresolved; translate supported legacy values or land an explicit, approved migration with regression coverage.
• Resolve merge risk (P1) - Existing native-evaluation jobs or fleet manifests that still use the legacy environment or manifest keys will fail before a benchmark starts.
• Resolve merge risk (P1) - The PR body leaves a fresh matched direct-versus-Code-Mode batch unchecked, so a real benchmark run would materially increase confidence in the revised scoring evidence path.
• Complete next step (P2) - A narrow compatibility repair is evident, but this member-authored PR must remain open for maintainer review of the upgrade contract.
• Improve patch quality - Translate or migrate legacy environment and fleet-plan tool-mode values with focused regression coverage.
• Improve patch quality - Attach redacted output from a fresh matched direct-versus-Code-Mode batch that passes trajectory validation.

Findings

• [P1] Preserve an upgrade path for retired tool-mode configuration — scripts/native_eval/run_job.py:370-374

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Patch scope	9 files; 1,815 additions and 952 deletions	Configuration, execution, trajectory reconstruction, and tests change together, increasing upgrade-review surface.
Legacy rejection coverage	2 focused tests added	The added tests intentionally lock in hard failures for both environment and fleet-plan legacy inputs.

Root-cause cluster

Relationship: fixed_by_candidate
Canonical: #61
Summary: This PR is the candidate implementation for the linked native-evaluation tool-mode issue, which remains open until a compatible repair lands.

Members:

• canonical: fix(native-eval): make OpenClaw Tool Search controls explicit #61 - The PR body explicitly identifies this open issue as the behavior the branch is intended to repair.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge-risk options

Maintainer options:

1. Add a compatibility translation (recommended)
   Map supported legacy environment and fleet values to the corresponding explicit tool modes, cover both paths with regression tests, and retain validation for unsupported values.
2. Approve a breaking migration
   If rejecting existing automation is intentional, document the operator migration and obtain explicit maintainer approval before landing.
3. Pause for a narrower repair
   Split the compatibility migration from the broader runner rewrite if the intended permanent configuration contract is not yet settled.

Technical review

Best possible solution:

Translate supported legacy tool-mode inputs to the explicit modes with focused regression coverage, preserve rejection for unsupported values, and attach a redacted matched direct-versus-Code-Mode run showing validated trajectory exports.

Do we have a high-confidence way to reproduce the issue?

Yes. Source establishes that a non-empty legacy environment value or legacy fleet-manifest key now raises before dispatch; focused regression tests can reproduce each path without a remote benchmark run.

Is this the best way to solve the issue?

No. Explicit modes are a reasonable target, but hard-failing established inputs without translation or an approved migration is not the narrowest safe repair.

Full review comments:

• [P1] Preserve an upgrade path for retired tool-mode configuration — scripts/native_eval/run_job.py:370-374
SHELLBENCH_OPENCLAW_TOOL_SEARCH_MODE now raises before a run starts, and the parallel fleet-plan key is rejected too. Existing native-evaluation automation can therefore stop on upgrade rather than receive the intended explicit mode. This concern was visible at the prior reviewed head and remains unresolved; translate supported legacy values or land an explicit, approved migration with regression coverage.
  Confidence: 0.99
  Late finding: first raised on code an earlier review cycle already covered.

Overall correctness: patch is incorrect
Overall confidence: 0.99

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label justifications:

• P2: This can stop or invalidate native-evaluation workflows, but the evidence does not show a core-runtime outage or security incident.
• merge-risk: 🚨 compatibility: The branch makes existing environment and fleet-manifest configuration fail rather than preserving an upgrade path.
• rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🌊 off-meta tidepool and patch quality is 🦪 silver shellfish.
• status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Not applicable: This member-authored PR is not subject to the external-contributor proof gate; however, the unchecked matched benchmark batch remains useful release-confidence evidence.

Evidence

What I checked:

• Legacy environment input now fails: The branch raises for any non-empty legacy environment key instead of translating supported values to the new explicit tool modes, so existing jobs can stop before execution. (scripts/native_eval/run_job.py:370, f5b31cb1260b)
• Legacy fleet-plan input now fails: The branch also rejects the legacy fleet manifest key and tells operators to replace it, but does not provide conversion or upgrade compatibility. (scripts/native_eval/fleet.py:447, f5b31cb1260b)
• Prior finding remains at the same head: The latest completed review identified this upgrade-path blocker at the same reviewed head; comparing that head to itself confirms no intervening change to the affected paths. (scripts/native_eval/run_job.py:370, f5b31cb1260b)
• Current main does not contain this repair: Current main lacks the new explicit tool-mode migration and validation code, so this work is neither implemented nor superseded there. (scripts/native_eval/fleet.py:429, 884dd1bb5511)
• Feature-history ownership: Current main's native-evaluation runner and trace work is attributed to Vincent Koc across the relevant file history. (scripts/native_eval/harnesses.py:358, 884dd1bb5511)

Likely related people:

• Vincent Koc: Authored the current-main native-evaluation runner and delegated-trace work, and authored the five commits on this branch. (role: recent area contributor; confidence: high; commits: 69f75c6629c4, 884dd1bb5511, f5b31cb1260b; files: scripts/native_eval/harnesses.py, scripts/native_eval/run_job.py, scripts/native_eval/fleet.py)

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (16 earlier review cycles; latest 8 shown)

• reviewed 2026-07-31T07:36:07.743Z sha f5b31cb :: needs maintainer review before merge. :: none
• reviewed 2026-07-31T10:30:27.123Z sha f5b31cb :: needs maintainer review before merge. :: none
• reviewed 2026-07-31T19:46:17.401Z sha f5b31cb :: needs maintainer review before merge. :: none
• reviewed 2026-08-01T00:25:04.802Z sha f5b31cb :: needs maintainer review before merge. :: none
• reviewed 2026-08-01T10:56:38.512Z sha f5b31cb :: needs maintainer review before merge. :: none
• reviewed 2026-08-01T12:21:15.559Z sha f5b31cb :: found issues before merge. :: [P1] Preserve an upgrade path for retired tool-mode configuration
• reviewed 2026-08-01T17:56:46.557Z sha f5b31cb :: found issues before merge. :: [P1] Preserve an upgrade path for retired tool-mode configuration
• reviewed 2026-08-01T19:16:46.445Z sha f5b31cb :: found issues before merge. :: [P1] Preserve an upgrade path for retired tool-mode configuration