fix: apply native reasoning effort to harnesses

[vincentkoc]

What does this PR do?

Propagates the native eval plan's reasoning effort into OpenClaw, Hermes,
Codex, and Claude Code client controls.

Why?

Fixes #53. Without this, a run can be labeled high while OpenClaw executes
with thinking disabled, invalidating cross-harness quality and efficiency
comparisons.

Changes

• carry reasoning_effort in RunSpec and the run manifest
• map effort to each native harness's supported control
• translate ShellBench's canonical xhigh to Claude Code's max
• preserve existing harness defaults when effort is unspecified
• test low, medium, high, and xhigh across all four harnesses

Tests

• 114 focused native runner, proxy, and fleet tests pass
• focused Ruff check passes
• fresh Codex autoreview reports no actionable findings
• all GitHub CI and CodeQL checks pass

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed July 30, 2026, 2:59 PM ET / 18:59 UTC.

ClawSweeper review

What this changes

Carries a native evaluation plan’s reasoning-effort setting through the run specification and manifest, then applies each harness’s supported OpenClaw, Hermes, Codex, or Claude Code control.

Merge readiness

⚠️ Ready for maintainer review - 2 items remain

Keep this member-authored PR open for maintainer review. The focused patch directly addresses the linked benchmark-validity bug, has no actionable code defect in the supplied diff, and needs confirmation against the installed native client versions before merge.

Priority: P1
Reviewed head: 887c2597c1e6ef7ad369cd9a72c800b8fb52ffeb
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🐚 platinum hermit (4/6)	A focused, well-covered parity repair with clean reported CI; the remaining confidence gap is external native-client compatibility rather than an identified patch defect.
Proof confidence	🌊 off-meta tidepool	Not applicable: This is a member-authored PR; the external-contributor real-behavior-proof gate does not apply, though native compatibility confirmation remains useful before merge.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Not applicable	Not applicable: This is a member-authored PR; the external-contributor real-behavior-proof gate does not apply, though native compatibility confirmation remains useful before merge.
Evidence reviewed	4 items	Linked behavior and intended fix: The PR explicitly targets the open canonical report that documents a planned high reasoning level being executed by OpenClaw with thinking disabled; its stated controls match that report’s expected OpenClaw, Hermes, Codex, and Claude Code mappings. Focused implementation and regression coverage: The branch changes four native-evaluation files: it adds reasoning effort to RunSpec and manifests, forwards it to all four harnesses, and adds parameterized coverage for four effort levels plus unspecified defaults. Review continuity: The previous completed review cycles examined the same head SHA and reported no actionable findings; this review finds no new branch change or concrete regression to add.
Findings	None	None.
Security	None	None.

How this fits together

ShellBench builds a native evaluation run plan, converts it into a harness command and per-run configuration, and records the selected settings in a manifest. The changed path ensures the planned reasoning level reaches each client so benchmark results reflect the intended comparison settings.

flowchart LR
  A[Evaluation plan] --> B[Run specification]
  B --> C[Harness command builder]
  B --> D[Run manifest]
  C --> E[OpenClaw Hermes Codex Claude Code]
  E --> F[Native evaluation run]
  D --> G[Benchmark comparison records]

Loading

Decision needed

Question	Recommendation
Should this member-authored PR merge after maintainers confirm that the installed OpenClaw, Hermes, Codex, and Claude Code clients accept and honor the mapped reasoning-effort values?	Verify and merge: Run a small native smoke evaluation for each harness and merge if its observed client behavior matches the run manifest.

Why: The patch is mechanically focused, but its correctness depends on four external client contracts that string-construction tests cannot fully prove in this checkout review.

Before merge

• Resolve merge risk (P1) - Each native harness accepts reasoning effort through an external CLI or configuration contract; maintainer confirmation with the repository’s installed client versions is still needed to rule out version-specific rejection or interpretation differences.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Patch scope	4 files affected; 121 added, 3 removed	The change is confined to native run construction, manifest recording, and focused regression coverage.
Harness-effort coverage	16 explicit harness/effort combinations	The parameterized test covers four canonical effort levels across all four native harnesses.

Merge-risk options

Maintainer options:

1. Verify native client compatibility before merge (recommended)
   Run one representative native evaluation per harness and confirm the accepted reasoning level matches the manifest and intended client control.
2. Accept the version-contract risk
   Merge based on the focused tests and current CI while accepting that installed client versions may differ in accepted reasoning-effort vocabulary.

Technical review

Best possible solution:

Confirm one real native run per harness at a non-default effort level, verify the manifest and client invocation agree, then merge the focused parity fix.

Do we have a high-confidence way to reproduce the issue?

No high-confidence live reproduction was executed in this review. The supplied linked report and branch diff make the original mismatch source-reproducible: a planned effort could diverge from the hard-coded OpenClaw thinking setting.

Is this the best way to solve the issue?

Yes, conditionally. Carrying the planned value through the shared run specification and mapping it at each harness boundary is the narrowest maintainable fix, provided native client compatibility is confirmed before merge.

AGENTS.md: unclear because the file could not be read completely.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label justifications:

• P1: Incorrect reasoning-level propagation can invalidate active cross-harness quality, latency, and efficiency comparisons.
• merge-risk: 🚨 compatibility: The patch passes canonical effort values into four external CLI or configuration interfaces whose accepted vocabulary can vary by installed version.
• rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: This is a member-authored PR; the external-contributor real-behavior-proof gate does not apply, though native compatibility confirmation remains useful before merge.

Evidence

What I checked:

• Linked behavior and intended fix: The PR explicitly targets the open canonical report that documents a planned high reasoning level being executed by OpenClaw with thinking disabled; its stated controls match that report’s expected OpenClaw, Hermes, Codex, and Claude Code mappings. (scripts/native_eval/harnesses.py:532, 887c2597c1e6)
• Focused implementation and regression coverage: The branch changes four native-evaluation files: it adds reasoning effort to RunSpec and manifests, forwards it to all four harnesses, and adds parameterized coverage for four effort levels plus unspecified defaults. (tests/test_native_eval_runner.py:681, 887c2597c1e6)
• Review continuity: The previous completed review cycles examined the same head SHA and reported no actionable findings; this review finds no new branch change or concrete regression to add. (scripts/native_eval/harnesses.py:817, 887c2597c1e6)
• Automated validation: Supplied PR check data reports successful Python 3.11/3.12 suites, CodeQL, and dispatch checks for the current branch. (887c2597c1e6)

Likely related people:

• vincentkoc: A repository member who authored both commits on this patch and the linked canonical report defining the affected native-evaluation behavior. (role: likely follow-up owner; confidence: medium; commits: 98301b240507, 887c2597c1e6; files: scripts/native_eval/harnesses.py, scripts/native_eval/models.py, scripts/native_eval/run_job.py)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Provide or perform one representative native-run confirmation for each harness, with sensitive endpoint and token details redacted.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (5 earlier review cycles)

• reviewed 2026-07-29T16:06:36.439Z sha 98301b2 :: needs maintainer review before merge. :: none
• reviewed 2026-07-29T16:20:31.880Z sha 887c259 :: needs maintainer review before merge. :: none
• reviewed 2026-07-29T16:27:52.992Z sha 887c259 :: needs maintainer review before merge. :: none
• reviewed 2026-07-29T21:18:42.277Z sha 887c259 :: needs maintainer review before merge. :: none
• reviewed 2026-07-30T07:39:25.377Z sha 887c259 :: needs maintainer review before merge. :: none