feat(native-eval): support OpenClaw candidate packages

[vincentkoc]

What does this PR do?

Adds exact local OpenClaw package artifacts to native evaluation campaigns.

Why?

ShellBench needs reproducible pre-publication baseline/candidate runs without
publishing temporary npm versions or mutating warmed machines.

Fixes #56

Changes

• add --openclaw-package-tarball campaign input and OpenClaw-only validation
• stage, upload, verify, and install the exact package artifact
• bind resume identity to package name, version, and SHA-256
• record candidate provenance in run manifests and research audits
• document the candidate workflow and failure modes

Exact proof

• Candidate tarball SHA:
  933616f90607b43970d2aaaf1de051a5fe8a751e514882f69c557d56e15755d2
• Exact c7a.8xlarge AWS run installed the package and exported three results.
• Exported manifests preserved package name openclaw, version 2026.7.2,
  artifact filename, and SHA-256.
• The run later exposed an independent Tool Search control bug tracked in fix(native-eval): make OpenClaw Tool Search controls explicit #61;
  that invalidates its score comparison, not the package staging proof.

Tests

• 108 focused native-eval tests pass

• Ruff passes on all touched Python files

• fresh Codex autoreview reports no actionable findings

• python -m pytest -q passes locally

• python -m ruff check clawbench app.py scripts tests passes locally, or the change is docs-only

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed August 1, 2026, 1:57 PM ET / 17:57 UTC.

ClawSweeper review

What this changes

The branch adds an OpenClaw-only --openclaw-package-tarball option that stages a local npm artifact for remote native-evaluation installation and records its identity in resume, run, and research-audit metadata.

Merge readiness

⛔ Blocked until stronger real behavior proof is added - 4 items remain

Keep this member-authored PR open. It remains the live implementation candidate for the requested native-evaluation workflow; its provenance design is focused, but a maintainer must explicitly confirm that fleet-controller access authorizes npm lifecycle execution from a supplied OpenClaw package before merge.

Priority: P2
Reviewed head: 3da8ec53b7964b64469dafd3090abcf318d942c8
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🦪 silver shellfish (2/6)	The patch and focused regression coverage are substantial, but inspectable live proof and an approved remote artifact-execution boundary are still required for merge confidence.
Proof confidence	🦪 silver shellfish (2/6)	Needs stronger real behavior proof before merge: The PR body claims a successful AWS run, but the supplied prepared media contains only bot button images; add redacted terminal output, logs, or exported artifacts that directly show staging, SHA-256 verification, installation, and matching provenance. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Patch quality	🐚 platinum hermit (4/6)	Security review found an item that needs attention.

Verification

Check	Result	Evidence
Real behavior	Needs proof	Needs stronger real behavior proof before merge: The PR body claims a successful AWS run, but the supplied prepared media contains only bot button images; add redacted terminal output, logs, or exported artifacts that directly show staging, SHA-256 verification, installation, and matching provenance. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed	6 items	Current main does not implement the requested workflow: A complete current-tree search found no --openclaw-package-tarball, openclaw_package, or candidate-package implementation on main, while the PR adds the feature across fleet, bootstrap, manifests, and audits. Candidate identity and resume safeguards: The PR verifies that the package is OpenClaw, binds package version and SHA-256 into campaign identity, rejects changed/missing candidate inputs on resume, stages the archive with mode 0600, and checks its SHA-256 after staging. Remote execution boundary: After checksum and package-metadata checks, bootstrap passes the supplied tarball to npm install on the leased host. This makes package lifecycle execution an intentional controller-access trust boundary, not merely an integrity check.
Findings	None	None.
Security	Needs attention	Confirm candidate artifact execution authority: The code verifies the candidate archive and then passes it to npm during bootstrap. Package lifecycle hooks can execute on the leased host, so controller access must explicitly authorize that path or installation must be constrained.

How this fits together

ShellBench's native-evaluation fleet controller stages runner and task inputs, then bootstraps leased hosts before dispatching benchmark jobs. This change adds a candidate OpenClaw artifact to that path and carries its identity through toolchain, run, and audit outputs.

flowchart LR
  A[Candidate package tarball] --> B[Fleet controller]
  B --> C[Campaign artifact staging]
  C --> D[Remote host bootstrap]
  D --> E[OpenClaw package install]
  E --> F[Run and toolchain manifests]
  F --> G[Research audit export]

Loading

Decision needed

Question	Recommendation
Should an operator who can submit a native-evaluation campaign be permitted to supply an OpenClaw npm tarball whose normal npm installation lifecycle runs on leased remote hosts?	Approve trusted controller submissions: Document that fleet-controller access authorizes a trusted OpenClaw candidate package and its normal npm installation lifecycle on leased evaluation hosts.

Why: The branch proves artifact identity through package metadata and SHA-256, but whether those identified bytes are authorized to execute during remote bootstrap is a maintainer security-policy decision.

Before merge

• Resolve security concern: Confirm candidate artifact execution authority - The code verifies the candidate archive and then passes it to npm during bootstrap. Package lifecycle hooks can execute on the leased host, so controller access must explicitly authorize that path or installation must be constrained.
• Resolve merge risk (P1) - A campaign-supplied npm tarball is installed during remote bootstrap; its lifecycle hooks can execute on leased hosts, so package identity verification does not itself establish artifact-execution authority.
• Resolve merge risk (P1) - The PR body reports an AWS run, but no inspectable runtime transcript, log, screenshot, recording, or exported artifact currently demonstrates the full after-fix staging, installation, and provenance path.

Findings

• [medium] Confirm candidate artifact execution authority — scripts/native_eval/bootstrap_beast.sh:94

Agent review details

Security

Needs attention: The branch has useful integrity checks, but a supplied npm artifact reaches remote installation and needs an explicit authorization boundary.

Review metrics

Metric	Value	Why it matters
Patch surface	10 files affected; 823 added, 23 removed	The feature spans controller staging, remote bootstrap, manifests, audits, documentation, and focused regression coverage.

Merge-risk options

Maintainer options:

1. Confirm trusted-controller semantics (recommended)
   Merge after maintainer-visible confirmation that a campaign submitter is trusted to provide an OpenClaw package whose npm installation lifecycle executes on leased hosts.
2. Constrain candidate admission
   Revise the install path to enforce a maintainer-approved candidate-artifact admission policy before bootstrap invokes npm on the archive.

Technical review

Best possible solution:

Confirm and document the trusted-controller model for OpenClaw candidate packages, or introduce an explicit artifact-admission mechanism before remote installation; preserve the branch's package identity, resume, manifest, and audit checks.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this PR adds a new native-evaluation capability rather than repairing a defined current-main failure.

Is this the best way to solve the issue?

Unclear: the identity and provenance design is a focused way to make candidate runs reproducible, but the remote package-execution boundary needs explicit maintainer approval before it is the best supported solution.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label justifications:

• P2: This is a bounded native-evaluation capability with meaningful benchmark-operator impact but no demonstrated user-facing outage.
• merge-risk: 🚨 security-boundary: The branch accepts a supplied package artifact and routes it through npm installation on leased hosts, where lifecycle code may execute.
• rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🐚 platinum hermit.
• status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The PR body claims a successful AWS run, but the supplied prepared media contains only bot button images; add redacted terminal output, logs, or exported artifacts that directly show staging, SHA-256 verification, installation, and matching provenance. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Evidence

Security concerns:

• [medium] Confirm candidate artifact execution authority — scripts/native_eval/bootstrap_beast.sh:94
  The code verifies the candidate archive and then passes it to npm during bootstrap. Package lifecycle hooks can execute on the leased host, so controller access must explicitly authorize that path or installation must be constrained.
  Confidence: 0.97

What I checked:

• Current main does not implement the requested workflow: A complete current-tree search found no --openclaw-package-tarball, openclaw_package, or candidate-package implementation on main, while the PR adds the feature across fleet, bootstrap, manifests, and audits. (scripts/native_eval/fleet.py:1848, 884dd1bb5511)
• Candidate identity and resume safeguards: The PR verifies that the package is OpenClaw, binds package version and SHA-256 into campaign identity, rejects changed/missing candidate inputs on resume, stages the archive with mode 0600, and checks its SHA-256 after staging. (scripts/native_eval/fleet.py:539, 3da8ec53b796)
• Remote execution boundary: After checksum and package-metadata checks, bootstrap passes the supplied tarball to npm install on the leased host. This makes package lifecycle execution an intentional controller-access trust boundary, not merely an integrity check. (scripts/native_eval/bootstrap_beast.sh:94, 3da8ec53b796)
• Feature ownership history: The existing native-evaluation fleet and bootstrap path appears to date to commit 69f75c6, and all visible history for the two central files is by Vincent Koc; the PR continues that area with three candidate-package commits. (scripts/native_eval/bootstrap_beast.sh:68, 69f75c6629c4)
• Real-run proof is not inspectable in supplied artifacts: The prepared media manifest contains only two Blacksmith promotional button images, which were inspected and do not show candidate staging, installation, checksum verification, or provenance output. (/home/runner/_work/clawsweeper/clawsweeper/artifacts/event/codex/proof-scratch/57/media-proof-manifest.json:1, 3da8ec53b796)
• Repository review policy: The full applicable parent-scoped AGENTS.md requires current real-behavior proof for code-bearing PRs and says maintainer-authored items are non-closeable; this PR's author association is MEMBER. (../AGENTS.md:1, 884dd1bb5511)

Likely related people:

• vincentkoc: Current-main history for the native fleet and bootstrap files is entirely attributed to this contributor, including the original native-evaluation implementation and this PR's candidate-package follow-ups. (role: feature introducer and recent area contributor; confidence: high; commits: 69f75c6629c4, 2cda5f84d5d4, 6e8fa399df99; files: scripts/native_eval/fleet.py, scripts/native_eval/bootstrap_beast.sh)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Add redacted terminal output, logs, or exported run artifacts showing tarball staging, checksum verification, installation, and matching run/audit provenance.
• Record maintainer confirmation of the trusted-controller execution boundary, or revise the branch to a narrowly approved artifact-admission path.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (14 earlier review cycles; latest 8 shown)

• reviewed 2026-07-31T04:05:43.536Z sha 3da8ec5 :: needs maintainer review before merge. :: none
• reviewed 2026-07-31T07:38:18.892Z sha 3da8ec5 :: needs maintainer review before merge. :: none
• reviewed 2026-07-31T10:30:50.778Z sha 3da8ec5 :: needs maintainer review before merge. :: none
• reviewed 2026-07-31T19:46:34.686Z sha 3da8ec5 :: needs maintainer review before merge. :: none
• reviewed 2026-08-01T00:25:47.301Z sha 3da8ec5 :: needs maintainer review before merge. :: none
• reviewed 2026-08-01T06:14:50.813Z sha 3da8ec5 :: needs maintainer review before merge. :: none
• reviewed 2026-08-01T10:56:42.938Z sha 3da8ec5 :: needs maintainer review before merge. :: none
• reviewed 2026-08-01T12:21:53.837Z sha 3da8ec5 :: needs maintainer review before merge. :: none