feat(native-eval): export normalized tool discovery telemetry

[vincentkoc]

Summary

Export normalized native-harness discovery telemetry separately from final tool
calls so ShellBench can compare deferred tool loading without treating missing
events as zero.

Motivation

The current research audit exports ATIF tool calls, but it loses harness-native
discovery detail. OpenClaw's tool_search_code result already contains catalog
size and search, describe, and call counts; the normalized CSV records only the
wrapper call. Codex native tool search exists in its runtime protocol, but
codex exec --json does not currently expose those events. Claude Code and
Hermes do not provide an equivalent event in the traces used by this runner.

This makes tool-search comparisons ambiguous: a missing count can mean zero
operations, unsupported discovery, or an exporter fidelity gap.

Proposed approach

• Add a discovery-event research table with search, describe, load, and
  call operations plus source harness, target ids, latency when available,
  success, and trace fidelity.
• Extract OpenClaw counters and selected calls from the structured
  tool_search_code result details.
• Mark Codex as unobservable until its native JSON stream exports ToolSearch
  items, and mark harnesses without a discovery surface as unsupported.
• Keep discovery operations separate from final tool calls and report runner
  SHA plus trajectory fidelity with every row.

Alternatives considered

Inferring discovery from wrapper names or recording absent events as zero was
rejected because both produce false cross-harness comparisons.

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed July 31, 2026, 12:05 AM ET / 04:05 UTC.

Summary
Keep open. This member-authored feature request has an active linked implementation candidate at #60, while maintainers still need to approve the durable cross-harness fidelity contract before the proposed telemetry becomes part of ShellBench’s research exports.

Reproducibility: not applicable. this requests a new research-export capability, not a failure of documented existing behavior. The supplied context clearly identifies the missing telemetry boundary and an active implementation candidate.

Root-cause cluster
Relationship: canonical
Canonical: #59
Summary: The linked pull request is the sole evidence-backed implementation candidate for this feature request and has not merged.

Members:

• fixed_by_candidate: feat(native-eval): export tool discovery telemetry #60 - It explicitly proposes the requested telemetry export, but it remains open and therefore cannot yet resolve this issue.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Ways to help us reproduce this

• Add a screenshot or short recording showing the behavior.
• Include the exact command, prompt, or workflow that triggered it.
• Add expected vs actual behavior.

Maintainer decision needed

Question	Recommendation
Should ShellBench publish a normalized discovery-event dataset with distinct observed, disabled, unsupported, and unobservable fidelity states?	Adopt an explicit fidelity contract: Approve a documented minimum schema that preserves unknown or unobservable discovery activity and review the linked implementation against it.

Why: The active pull request can implement the proposal, but only maintainers can choose whether these cross-harness research semantics should become a stable exported contract.

Next step
The linked pull request already owns the implementation path; maintainer approval of the telemetry and fidelity contract is the remaining action.

Review details

Best possible solution:

Approve a small documented discovery-event contract that preserves unknown or unobservable data rather than fabricating zeros, then review the linked pull request against that contract.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this requests a new research-export capability, not a failure of documented existing behavior. The supplied context clearly identifies the missing telemetry boundary and an active implementation candidate.

Is this the best way to solve the issue?

Unclear: separating discovery events from final calls is a sound way to avoid false zero counts, but maintainers must first approve the durable fidelity vocabulary and comparison semantics.

AGENTS.md: unclear because the file could not be read completely.

Remaining risk / open question:

• A normalized fidelity vocabulary can become a long-lived research contract; an imprecise distinction between disabled, unsupported, and unobservable traces could mislead cross-harness comparisons.
• Closing this issue before the linked pull request is accepted would conceal an unresolved implementation and product-methodology decision.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Label changes

Label justifications:

• P3: This is a non-urgent research-export feature proposal rather than a current user-facing regression.
• impact:other: The requested change affects benchmark-data interpretation and cross-harness comparability, outside the more specific owned impact classes.

Evidence reviewed

What I checked:

• Open implementation candidate: The issue is explicitly linked to an open pull request that states it resolves this request. Its head is based on current main, so the requested code is not yet part of the reviewed default-branch state. (15dce3354406)
• Main-branch boundary: The linked pull request’s base SHA is the supplied current main SHA, while its implementation is on a distinct head SHA; this supports keeping the issue open until the candidate is reviewed and merged. (884dd1bb5511)
• Proposal requires methodology direction: The request defines new exported research concepts—separate discovery operations plus observed, disabled, unsupported, and unobservable fidelity states—rather than repairing an established existing contract.

Likely related people:

• vincentkoc: Authored this member-owned feature request and the active linked pull request that specifies the proposed telemetry schema and fidelity semantics. (role: proposal and linked implementation author; confidence: medium; commits: 15dce3354406; files: scripts/native_eval/research_audit.py, tests/test_native_eval_research_audit.py)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.