Skip to content

fix(eval): separate parity from score eligibility - #47

Merged
vincentkoc merged 1 commit into
mainfrom
fix/native-parity-reporting
Jul 29, 2026
Merged

fix(eval): separate parity from score eligibility#47
vincentkoc merged 1 commit into
mainfrom
fix/native-parity-reporting

Conversation

@vincentkoc

Copy link
Copy Markdown
Member

What does this PR do?

Reports scoped Harbor parity independently from native leaderboard eligibility, so complete native runs remain scoreable without an unsupported parity claim.

Why?

The scoped-parity change in #46 accidentally made parity validation a hard eligibility gate. Reprocessing the repaired 93-run matrix then validated every archive but selected 0/93 runs, all with parity_not_validated. The ShellBench matrix contract gates eligibility on coverage, infrastructure health, harness-wide failures, model identity, and trace completeness; parity is separate audit metadata.

Changes

  • remove parity validation from native score eligibility
  • retain per-run parity status and scope
  • add pair-level parity counts and unvalidated run labels
  • show parity coverage separately in the cleaned leaderboard
  • update parity regression expectations

Tests

  • PYTHONPATH=. .venv/bin/pytest -q tests/test_native_eval*.py (99 passed)

  • .venv/bin/ruff check scripts/native_eval tests/test_native_eval*.py

  • repaired seed replay: 93 selected, 93 eligible, 0 parity-validated, no exclusion reasons

  • python -m pytest -q equivalent affected native suite passes locally

  • python -m ruff check clawbench app.py scripts tests affected native surfaces pass locally

@vincentkoc
vincentkoc marked this pull request as ready for review July 29, 2026 03:09
@vincentkoc
vincentkoc requested a review from a team as a code owner July 29, 2026 03:09
@vincentkoc
vincentkoc merged commit 9afe6a4 into main Jul 29, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant