Skip to content

fix(eval): stabilize native harness results and traces - #46

Merged
vincentkoc merged 28 commits into
mainfrom
fix/native-runner-integration
Jul 29, 2026
Merged

fix(eval): stabilize native harness results and traces#46
vincentkoc merged 28 commits into
mainfrom
fix/native-runner-integration

Conversation

@vincentkoc

Copy link
Copy Markdown
Member

Summary

Consolidates the still-unmerged child work from #43, #44, and #45 onto the landed native runner from #42. The original stacked branches cannot be force-updated under repository protection after #42 was squash-merged.

Changes

  • preserve explicit GPT reasoning effort and add GPT-5.6 Luna/Terra catalog IDs
  • preserve real OpenClaw, Hermes, and Codex trajectories
  • isolate agent and judge reasoning configuration
  • prevent OpenClaw onboarding bootstrap contamination
  • recover active/degraded Crabbox leases and retain zero-result failure checkpoints
  • support targeted canonical repair overlays and canonical task identities
  • scope parity claims to the exact validated route; legacy Codex GPT-5.5 compatibility is limited to the fixed 20-task calibration
  • emit original-versus-repaired score sensitivity reports

Validation

  • pytest -q tests/test_native_eval*.py: 99 passed
  • focused parity/repair suite: 91 passed
  • Ruff passes on native runner and test surfaces
  • git diff --check passes
  • all 28 replayed commits carry valid signatures
  • prior 93-run campaign: 10,788/10,788 task results, zero infra rows
  • the 117-run low/medium/high matrix extension is running separately with verified periodic checkpoints

Stack cleanup

This PR supersedes #43, #44, and #45 without discarding their commit history. Result publications remain model-scoped in ShellBench/public-tasks.

@vincentkoc
vincentkoc marked this pull request as ready for review July 29, 2026 02:54
@vincentkoc
vincentkoc requested a review from a team as a code owner July 29, 2026 02:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant