Native harnesses do not apply planned reasoning effort consistently

[vincentkoc]

Problem

Native eval plans record a reasoning effort, and the LiteLLM proxy enforces it upstream, but the harness clients do not consistently receive the same setting.

The clearest mismatch is OpenClaw: a reasoning_effort: high run still invokes openclaw agent --thinking off. This makes cross-harness comparisons confounded because the provider request and the agent runtime disagree about reasoning mode.

Expected behavior

Each native harness should receive the plan's reasoning effort through its supported client control:

• OpenClaw: --thinking
• Hermes: agent.reasoning_effort
• Codex: model_reasoning_effort
• Claude Code: --effort

When the plan omits reasoning effort, each harness should retain its existing default.

Evidence

Observed in the July 29, 2026 OpenClaw r0 trace:

• run index: reasoning_effort: high
• OpenClaw command: --thinking off
• trace metadata: thinkLevel: off

This should be fixed before using token, latency, or quality deltas for harness comparisons.

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed August 1, 2026, 7:08 PM ET / 23:08 UTC.

Summary
Keep this member-authored issue open: current main still passes planned reasoning effort to the proxy and run metadata but does not carry it into native harness commands. The linked open repair PR, #54, explicitly owns the bounded fix, so opening a parallel repair would duplicate work. Likely related people: Vincent Koc is the high-confidence routing candidate from the native-evaluation history.

Reproducibility: yes. —current source provides a high-confidence reproduction path: a planned value reaches fleet environment variables and proxy configuration, while the native command builder still emits OpenClaw --thinking off and no equivalent control for the other harnesses.

Next step
The explicitly linked open PR #54 already owns this focused repair, and this member-authored issue must remain open for maintainer review until that work lands.

Review details

Best possible solution:

Review and land the focused linked PR after it demonstrates redacted after-fix native command or trace evidence for planned effort and confirms omitted effort keeps each harness default.

Do we have a high-confidence way to reproduce the issue?

Yes—current source provides a high-confidence reproduction path: a planned value reaches fleet environment variables and proxy configuration, while the native command builder still emits OpenClaw --thinking off and no equivalent control for the other harnesses.

Is this the best way to solve the issue?

Yes—the linked PR’s stated approach is the narrow maintainable repair because it propagates the existing plan value through the run specification and maps it at the established per-harness command boundary without adding a new user-facing mode.

AGENTS.md: found and applied where relevant.

Remaining risk / open question:

• The remaining repair is in an open PR whose supplied context is labeled as needing real behavior proof, so benchmark comparability remains unverified until maintainers review the current head and acceptable after-fix evidence is added.
• Deeper line-level provenance is partially unavailable in this partial checkout because missing promisor objects cannot be fetched; this does not change the source-proven current behavior or the linked-PR routing.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Label changes

Label justifications:

• P1: A planned high-effort run can execute native harnesses with incompatible reasoning controls, undermining active cross-harness benchmark comparisons.
• impact:other: The concrete impact is invalid benchmark comparability, which does not fit the more specific owned impact categories.

Evidence reviewed

What I checked:

• Current source preserves the mismatch: The fleet dispatches the planned reasoning effort as SHELLBENCH_REASONING_EFFORT and the proxy applies it upstream, but RunSpec has no reasoning-effort field and the command builder cannot receive that value. (scripts/native_eval/models.py:24, 884dd1bb5511)
• OpenClaw is explicitly forced off: Current main invokes OpenClaw with --thinking off; Hermes, Codex, and Claude Code command paths likewise have no per-run reasoning-effort control. (scripts/native_eval/harnesses.py:619, 884dd1bb5511)
• Regression coverage currently asserts the incorrect OpenClaw setting: The native command test on current main expects --thinking off, confirming the reported behavior remains intentional in the checked-out code rather than already fixed. (tests/test_native_eval_runner.py:645, 884dd1bb5511)
• Linked implementation candidate: The supplied GitHub context shows the open, mergeable PR fix: apply native reasoning effort to harnesses #54 uses closing syntax for this issue and proposes RunSpec propagation, per-harness mapping, default preservation, and focused coverage. (887c2597c1e6)
• Recent area history: Recent commits touching the native harness command path are by Vincent Koc, including the current main commit and the preceding native-evaluation stabilization work. (scripts/native_eval/harnesses.py:619, 884dd1bb5511)
• Maturity score unavailable: The required maturity helper could not find qa/maturity-scores.yaml in this checkout, so no stable-maturity label is supportable.

Likely related people:

• Vincent Koc: Recent native harness commits on current main and the preceding stabilization commit are authored by Vincent Koc; the linked open implementation is also by the same member. (role: recent area contributor; confidence: high; commits: 884dd1bb5511, b9acd9f7a010, 69f75c6629c4; files: scripts/native_eval/harnesses.py, scripts/native_eval/models.py, scripts/native_eval/fleet.py)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.