Run Hermes adapter through local harness

[afrog33k]

Summary

• Wire registered non-OpenClaw adapters through the generic harness lifecycle instead of rejecting them as known-but-unwired.
• Add Hermes local adapter configuration from environment and a local verification shim for file/execution checks.
• Prepend explicit staged-workspace guidance to Hermes MiniSWERunner prompts so local runs inspect the task workspace first.

Validation

• /Users/reckon/miniconda3/bin/python -m pytest tests/test_harness.py tests/test_hermes_adapter.py -q -> 21 passed
• /Users/reckon/miniconda3/bin/python -m pytest -q -> 318 passed, 5 skipped
• Local Hermes rerun: t2-msg-summarize-thread improved from 0.63704 to 1.000 with the workspace-guidance patch.

Notes

• This keeps OpenClaw behavior intact and routes adapter != "openclaw" through the adapter lifecycle.
• Memory/session/cron/browser checks still need adapter-aware verification before broad non-file task sweeps.

[clawsweeper]

Codex review: needs changes before merge. Reviewed July 31, 2026, 12:05 AM ET / 04:05 UTC.

ClawSweeper review

What this changes

This PR lets registered local adapters such as Hermes run through ShellBench’s generic benchmark harness, adds local file/execution verification support, and prepends task-workspace guidance to Hermes prompts.

Merge readiness

⚠️ Needs maintainer review before merge - 4 items remain

Keep this PR open: it has useful real-run evidence and a focused integration goal, but local state assertions still go through the gateway-only verification client instead of the adapter verifier. That makes memory, session, and cron checks fail or misreport for Hermes/local runs, so the harness cannot yet safely support the broader non-file workload it enables.

Priority: P2
Reviewed head: 67ff264005ab6b55679299ba4e7fcd7cbb1276bf

Review scores

Measure	Result	What it means
Overall readiness	🦐 gold shrimp (3/6)	Useful local-run proof and focused tests support the proposal, but a known P1 verification-path defect prevents reliable broader local-adapter scoring.
Proof confidence	🐚 platinum hermit (4/6)	Sufficient (live_output): The PR body supplies after-fix local Hermes benchmark output and reported test results, and the timeline records proof: sufficient; the remaining merge blocker is functional coverage, not missing proof.
Patch quality	🦐 gold shrimp (3/6)	1 actionable review finding remain.

Verification

Check	Result	Evidence
Real behavior	Verified	Sufficient (live_output): The PR body supplies after-fix local Hermes benchmark output and reported test results, and the timeline records proof: sufficient; the remaining merge blocker is functional coverage, not missing proof.
Evidence reviewed	4 items	Repeated current-head blocker: The supplied follow-up review records the same P1 finding against current head 67ff264: shared memory, session, and cron assertions still use _LocalVerificationClient rather than adapter-aware verify_state_query. Patch design exposes gateway-only failure path: The PR adds _LocalVerificationClient specifically for non-gateway local adapter runs, but its RPC and agent-file methods raise RuntimeError. The PR description also acknowledges that memory/session/cron/browser checks still need adapter-aware verification. Positive real behavior evidence: The PR body reports a local Hermes rerun of t2-msg-summarize-thread improving from 0.63704 to 1.000 and records focused and full pytest runs; the timeline also carries proof: sufficient.
Findings	1 actionable finding	[P1] Route local state queries through the adapter verifier
Security	None	None.

How this fits together

ShellBench turns benchmark tasks into adapter runs and then evaluates the resulting workspace, execution output, and agent state. This PR changes the local-adapter path between task execution and verification so Hermes can use the same harness lifecycle as OpenClaw without a gateway.

flowchart LR
  A[Benchmark task] --> B[Harness lifecycle]
  B --> C{Selected adapter}
  C -->|OpenClaw| D[Gateway client]
  C -->|Hermes or local adapter| E[Adapter context]
  E --> F[Workspace and execution results]
  D --> G[State verification]
  E --> G
  F --> H[Benchmark score]
  G --> H

Loading

Before merge

• Route local state queries through the adapter verifier (P1) - For non-OpenClaw runs, the new local verification client raises for gateway methods, while shared memory/session/cron assertions still use that client instead of verify_state_query. As a result, an adapter that supports those checks can be reported as failing; dispatch state queries to the selected adapter and keep the gateway path for OpenClaw.
• Resolve merge risk (P1) - Merging now would make local Hermes runs appear broadly supported while memory, session, and cron assertions still use a gateway-only client and can fail independently of the adapter’s actual capability.
• Resolve merge risk (P1) - The generic local-adapter lifecycle is compatibility-sensitive: adapters that implement state verification may be scored differently or incorrectly until the harness consistently selects the adapter verifier.
• Complete next step (P2) - A narrow mechanical repair remains: use the selected local adapter’s state verifier for shared state assertions while preserving the OpenClaw gateway path.

Findings

• [P1] Route local state queries through the adapter verifier — clawbench/harness.py:55

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Changed surface	4 files; 350 additions, 17 deletions	The change alters the shared harness lifecycle plus Hermes prompting and tests, so verification behavior affects more than one adapter file.
Validation scope	21 focused tests and 318 full-suite tests reported	The PR includes useful automated validation, but it does not cover the remaining local state-query path.

Merge-risk options

Maintainer options:

1. Route local state checks through adapters (recommended)
   Update the local harness path to call the selected adapter’s state verifier, retaining gateway verification for OpenClaw, then add focused capability tests.
2. Limit the feature to file and execution tasks
   Land only if the harness explicitly prevents local adapters from running state-query tasks until adapter-aware verification is implemented.
3. Pause the generic lifecycle change
   Pause this PR if maintainers do not want partial local-adapter support or a task-capability boundary in the harness.

Copy recommended automerge instruction

@clawsweeper automerge

Special instructions:
Route local state assertions through the selected adapter verifier while retaining the OpenClaw gateway path; add supported and capability-missing state-query coverage.

Technical review

Best possible solution:

Keep OpenClaw on its gateway verification path, but route local adapter state queries through each adapter’s verify_state_query method and add coverage for both supported and explicitly missing capabilities before expanding Hermes task sweeps.

Do we have a high-confidence way to reproduce the issue?

Yes, from source: run a local adapter task containing memory, session, or cron state assertions and the shared path reaches the gateway-only local verification client rather than the adapter verifier. The supplied review history identifies this against the unchanged current PR head.

Is this the best way to solve the issue?

No. The generic lifecycle is the right direction, but local adapter state queries must be dispatched to the adapter verifier rather than failing through a gateway-only shim.

Full review comments:

• [P1] Route local state queries through the adapter verifier — clawbench/harness.py:55
  For non-OpenClaw runs, the new local verification client raises for gateway methods, while shared memory/session/cron assertions still use that client instead of verify_state_query. As a result, an adapter that supports those checks can be reported as failing; dispatch state queries to the selected adapter and keep the gateway path for OpenClaw.
  Confidence: 0.96

Overall correctness: patch is incorrect
Overall confidence: 0.93

AGENTS.md: unclear because the file could not be read completely.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label changes:

• add merge-risk: 🚨 session-state: Session and memory assertions can be evaluated through a gateway-only client rather than the adapter’s state verifier.
• add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🐚 platinum hermit and patch quality is 🦐 gold shrimp.
• remove rating: 🧂 unranked krab: Current PR rating is rating: 🦐 gold shrimp, so this older rating label is no longer current.

Label justifications:

• P2: This is a bounded harness integration defect that blocks reliable non-OpenClaw benchmark coverage but does not indicate a current emergency.
• merge-risk: 🚨 compatibility: The PR changes the generic adapter execution and verification contract for registered non-OpenClaw adapters.
• merge-risk: 🚨 session-state: Session and memory assertions can be evaluated through a gateway-only client rather than the adapter’s state verifier.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🐚 platinum hermit and patch quality is 🦐 gold shrimp.
• status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (live_output): The PR body supplies after-fix local Hermes benchmark output and reported test results, and the timeline records proof: sufficient; the remaining merge blocker is functional coverage, not missing proof.
• proof: sufficient: Contributor real behavior proof is sufficient. The PR body supplies after-fix local Hermes benchmark output and reported test results, and the timeline records proof: sufficient; the remaining merge blocker is functional coverage, not missing proof.

Evidence

Acceptance criteria:

• [P1] python -m pytest tests/test_harness.py tests/test_hermes_adapter.py -q.
• [P1] python -m pytest -q.

What I checked:

• Repeated current-head blocker: The supplied follow-up review records the same P1 finding against current head 67ff264: shared memory, session, and cron assertions still use _LocalVerificationClient rather than adapter-aware verify_state_query. (clawbench/harness.py:55, 67ff264005ab)
• Patch design exposes gateway-only failure path: The PR adds _LocalVerificationClient specifically for non-gateway local adapter runs, but its RPC and agent-file methods raise RuntimeError. The PR description also acknowledges that memory/session/cron/browser checks still need adapter-aware verification. (clawbench/harness.py:49, 67ff264005ab)
• Positive real behavior evidence: The PR body reports a local Hermes rerun of t2-msg-summarize-thread improving from 0.63704 to 1.000 and records focused and full pytest runs; the timeline also carries proof: sufficient. (clawbench/adapters/hermes.py:477, 67ff264005ab)
• Policy inspection unavailable: A read-only shell inspection could not start because the sandbox failed before executing the command, so AGENTS.md and local git provenance could not be independently read in this review.

Likely related people:

• afrog33k: Authored both commits on the submitted branch and documented the remaining adapter-aware verification gap; no current-main ownership history could be inspected because the read-only command sandbox failed to start. (role: proposed integration author; confidence: low; commits: ba220dc7821b, 67ff264005ab; files: clawbench/harness.py, clawbench/adapters/hermes.py)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Route local state assertions through the adapter verifier and retain the OpenClaw gateway path.
• Add focused tests covering supported and capability-missing local state queries.
• Post a redacted live run covering a state-query task after the repair; if automatic review does not rerun, ask a maintainer to comment @clawsweeper re-review.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (12 earlier review cycles; latest 8 shown)

• reviewed 2026-07-07T21:17:21.869Z sha 67ff264 :: needs changes before merge. :: [P1] Route adapter state queries through the adapter verifier
• reviewed 2026-07-16T11:28:59.523Z sha 67ff264 :: needs changes before merge. :: [P1] Route adapter state queries through the adapter verifier
• reviewed 2026-07-29T21:18:43.773Z sha 67ff264 :: needs changes before merge. :: [P1] Route state queries through the adapter verifier
• reviewed 2026-07-29T23:18:18.207Z sha 67ff264 :: needs changes before merge. :: [P1] Route state queries through the adapter verifier
• reviewed 2026-07-30T07:39:10.929Z sha 67ff264 :: needs changes before merge. :: [P1] Route state queries through the adapter verifier
• reviewed 2026-07-30T18:59:34.331Z sha 67ff264 :: needs changes before merge. :: [P1] Route state queries through the adapter verifier
• reviewed 2026-07-31T01:05:39.162Z sha 67ff264 :: needs changes before merge. :: [P1] Route state queries through the adapter verifier
• reviewed 2026-07-31T01:19:00.005Z sha 67ff264 :: needs changes before merge. :: [P1] Route local state queries through the adapter verifier