feat(native-eval): export tool discovery telemetry

[vincentkoc]

What does this PR do?

Exports native-harness tool discovery operations separately from final tool
calls and records the trace fidelity needed to interpret missing events.

Why?

Fixes #59.

The existing research audit records tool_search_code as one wrapper call but
drops OpenClaw's embedded catalog and search/describe/call counters. It also
cannot distinguish disabled discovery from unsupported or unobservable native
events, which makes cross-harness comparisons produce fake zeros.

Changes

• add discovery_events.csv with normalized search, describe, and call counts
• extract OpenClaw code-bridge telemetry and structured Tool Search controls
• report disabled, observed, supported-but-unused, unobservable, and unsupported states
• add runner commit and trace fidelity to each task inventory row
• leave operation success unknown unless the trace proves it

Tests

• python -m pytest -q tests/test_native_eval_research_audit.py
• python -m ruff check scripts/native_eval/research_audit.py tests/test_native_eval_research_audit.py
• re-exported the completed released-off ablation artifacts and verified
  all three tasks report the expected runner SHA, session fidelity,
  disabled discovery state, and zero discovery operations

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed July 30, 2026, 2:59 PM ET / 18:59 UTC.

ClawSweeper review

What this changes

The PR adds normalized discovery-event CSV exports, OpenClaw tool-search telemetry extraction, trace-fidelity states, and runner-commit metadata to native-eval research audits.

Merge readiness

⚠️ Ready for maintainer review - 3 items remain

Keep this PR open for maintainer review. It is the active implementation candidate for the linked ShellBench telemetry request, but it expands research-export CSV schemas and needs an explicit compatibility decision for existing parsers and archived-data tooling before merge.

Priority: P3
Reviewed head: 15dce335440688dd25fa45c28495e3fc0daa75a6
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🦐 gold shrimp (3/6)	The patch has focused tests and a coherent scope, but maintainer-visible export-schema compatibility direction is still needed.
Proof confidence	🌊 off-meta tidepool	Not applicable: The PR author is a repository MEMBER, so the external-contributor real-behavior-proof gate does not apply; the PR body nevertheless reports a completed-artifact re-export check.
Patch quality	🦐 gold shrimp (3/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Not applicable	Not applicable: The PR author is a repository MEMBER, so the external-contributor real-behavior-proof gate does not apply; the PR body nevertheless reports a completed-artifact re-export check.
Evidence reviewed	5 items	Active implementation candidate: The PR body states that it addresses the linked native-eval discovery-telemetry request, and the related issue remains open with this PR marked as its linked implementation path. Export schema expansion: The branch adds task-inventory fields including runner commit, trace fidelity, discovery status, and discovery-event count, and introduces a separate discovery-events CSV schema. Focused regression coverage: The branch adds tests for observed-but-unused telemetry, unscoped zero-count telemetry, normalization, and unknown operation success rather than changing unrelated subsystems.
Findings	None	None.
Security	None	None.

How this fits together

ShellBench's native-eval research audit reads completed harness trajectories and exports CSV tables for later analysis. This PR parses discovery telemetry from those traces and emits it alongside task inventory, tool-call, usage, and model-identity data for downstream research consumers.

flowchart LR
  A[Completed native harness trajectories] --> B[Research audit exporter]
  B --> C[Trace and tool telemetry parsing]
  C --> D[Discovery operation normalization]
  D --> E[Trace fidelity classification]
  E --> F[Task inventory CSV]
  E --> G[Discovery events CSV]
  F --> H[Research analysis consumers]
  G --> H

Loading

Decision needed

Question	Recommendation
Should ShellBench treat research-audit CSV output as an explicitly additive, forward-compatible schema, including the new discovery-events table and task-inventory columns?	Adopt additive compatibility: Document that consumers must ignore unknown columns and tolerate additional CSV tables, then add focused compatibility coverage for that contract.

Why: The implementation is coherent, but only maintainers can decide whether downstream research tooling must tolerate additive columns and new tables or whether exports require versioning or migration support.

Before merge

• Resolve merge risk (P1) - Existing research parsers, notebooks, and archive-processing scripts may assume the exact current task-inventory header or CSV set; adding columns and a new table needs a stated additive-schema and upgrade policy before merge.
• Complete next step (P2) - A maintainer must choose the supported research-export compatibility contract before any narrow follow-up work can be safely defined.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Patch scope	2 files affected; 815 added, 3 removed	The change is narrowly scoped to one exporter and its regression tests, but materially expands its public CSV output.
Export surface	1 new CSV table; 4 task-inventory fields added	These additions are the compatibility-sensitive behavior that downstream research consumers must accept.

Root-cause cluster

Relationship: fixed_by_candidate
Canonical: #59
Summary: This PR is the active candidate implementation for the open ShellBench issue requesting normalized native-harness discovery telemetry.

Members:

• canonical: feat(native-eval): export normalized tool discovery telemetry #59 - The issue describes the same discovery-event export, OpenClaw counter extraction, trace-fidelity states, and runner metadata targeted by this PR.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge-risk options

Maintainer options:

1. Define additive CSV compatibility before merge (recommended)
   Document the supported consumer contract and add a focused compatibility check so existing parsers do not fail on the additional fields and table.
2. Accept a breaking research-export revision
   Land the new schema as an intentional consumer migration after maintainers identify the affected archive and notebook users.
3. Pause for schema ownership
   Defer the PR if no maintainer is ready to own the downstream export-contract decision.

Technical review

Best possible solution:

Adopt a documented additive export-schema contract, then retain the normalized discovery table and metadata with compatibility coverage for current CSV consumers.

Do we have a high-confidence way to reproduce the issue?

Not applicable as a defect reproduction: this is a new research-export capability. The supplied PR body describes an after-fix re-export of three completed artifacts, but the review context does not include the actual resulting CSV or validation transcript.

Is this the best way to solve the issue?

Unclear. Separating discovery operations from final tool calls is a maintainable analysis design, but the export compatibility contract must be chosen before this becomes the supported schema.

AGENTS.md: unclear because the file could not be read completely.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label justifications:

• P3: This is a research-export enhancement with no evidence of a broken user-facing runtime workflow.
• merge-risk: 🚨 compatibility: The PR adds task-inventory columns and a discovery-events CSV table that may affect consumers expecting the prior export schema.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🌊 off-meta tidepool and patch quality is 🦐 gold shrimp.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The PR author is a repository MEMBER, so the external-contributor real-behavior-proof gate does not apply; the PR body nevertheless reports a completed-artifact re-export check.

Evidence

What I checked:

• Active implementation candidate: The PR body states that it addresses the linked native-eval discovery-telemetry request, and the related issue remains open with this PR marked as its linked implementation path. (15dce3354406)
• Export schema expansion: The branch adds task-inventory fields including runner commit, trace fidelity, discovery status, and discovery-event count, and introduces a separate discovery-events CSV schema. (scripts/native_eval/research_audit.py:33, 15dce3354406)
• Focused regression coverage: The branch adds tests for observed-but-unused telemetry, unscoped zero-count telemetry, normalization, and unknown operation success rather than changing unrelated subsystems. (tests/test_native_eval_research_audit.py:7, 15dce3354406)
• Recent feature history: The supplied PR history identifies two native-eval commits: the initial telemetry export and a follow-up counter-normalization fix. The current head is the follow-up commit. (scripts/native_eval/research_audit.py:89, 15dce3354406)
• Repository inspection limitation: The review environment rejected read-only local shell commands before repository files could be opened, so no current-main source or AGENTS.md confirmation could be established beyond the supplied PR context.

Likely related people:

• vincentkoc: Authored the linked ShellBench discovery-telemetry request and both commits in the proposed native-eval telemetry implementation. (role: recent native-eval contributor; confidence: medium; commits: 7eefe9fe0c3b, 15dce3354406; files: scripts/native_eval/research_audit.py, tests/test_native_eval_research_audit.py)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Decide and document the additive-schema or schema-versioning contract for research-audit consumers.
• Add focused compatibility coverage once the supported CSV-consumer contract is chosen.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (5 earlier review cycles)

• reviewed 2026-07-29T18:07:40.062Z sha 7eefe9f :: needs maintainer review before merge. :: none
• reviewed 2026-07-29T20:33:51.773Z sha 15dce33 :: needs maintainer review before merge. :: none
• reviewed 2026-07-29T21:18:45.534Z sha 15dce33 :: needs maintainer review before merge. :: none
• reviewed 2026-07-29T23:18:15.364Z sha 15dce33 :: needs maintainer review before merge. :: none
• reviewed 2026-07-30T08:38:34.499Z sha 15dce33 :: needs maintainer review before merge. :: none