feat(skill-scan): add LLM model evaluation harness - #913
Open
danbarr wants to merge 3 commits into
Open
Conversation
Tooling to benchmark candidate LLM models for skill-security-scan against the skills with the worst churn/latency history (#904): - bench_models.py: runs the CI scan pipeline (run_scan.py unchanged) N times per model per skill at the pinned spec.ref, measuring scan latency, blocking churn under current allowlists, HIGH+ noise with allowlists disabled, and LLM-finding stability (pairwise Jaccard). Supports per-provider API keys, resume after interruption, and the scanner's consensus-runs knob. - mine_ci_baseline.py: reconstructs the production-model baseline from retained CI scan artifacts instead of re-running paid scans; groups by scanned spec.ref, drops cache-hit duplicates, and records scanner versions so cross-version noise is not misread as model noise. - merge_report.py: merges candidate summaries and the mined baseline into one comparison table, filtering baseline scans to the scanner version the candidates ran on. Results land in results/ (gitignored); API keys come from .env in this directory (also gitignored). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run scanner evaluation fixtures through the production scan wrapper so model comparisons include the LLM and meta analyzers. Record the initial Sonnet 4.6 versus Terra evaluation and clarify that finding counts require human review. Signed-off-by: Dan Barr <6922515+danbarr@users.noreply.github.com>
Contributor
🛡️ Skill Security Scan Results✅ agent-observability-eval-bootstrap
Summary: Scanned 1 skill(s), all passed security checks. ✅ |
Signed-off-by: Dan Barr <6922515+danbarr@users.noreply.github.com>
danbarr
added a commit
that referenced
this pull request
Aug 27, 2026
GPT-5.6 Terra rejects any explicit temperature value, and LiteLLM's drop_params doesn't catch this for every routing path (see #909). Auto-derive SKILL_SCANNER_LLM_TEMPERATURE=none when the configured model name contains gpt-5, matching the workaround already validated in the eval harness (scripts/skill-scan/eval/bench_models.py, #913).
danbarr
added a commit
that referenced
this pull request
Aug 27, 2026
* ci: use dedicated secret for skill scanner LLM key Per #909, the skill scanner moves to openai/gpt-5.6-terra while the MCP scanner stays on Anthropic, so they need separate provider keys. Switches build-skills.yml to the new SKILL_SCANNER_LLM_API_KEY repo secret instead of the shared MCP_SCANNER_LLM_API_KEY. * ci: omit LLM temperature for GPT-5.x reasoning models GPT-5.6 Terra rejects any explicit temperature value, and LiteLLM's drop_params doesn't catch this for every routing path (see #909). Auto-derive SKILL_SCANNER_LLM_TEMPERATURE=none when the configured model name contains gpt-5, matching the workaround already validated in the eval harness (scripts/skill-scan/eval/bench_models.py, #913). --------- Co-authored-by: Dan Barr <6922515+danbarr@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
run_scan.py, ensuring the LLM and meta analyzers are includedWhy
The skill security scan has experienced nondeterministic finding churn and long tail latency. This tooling makes future model comparisons repeatable across blocking behavior, raw HIGH+ findings, LLM finding-set stability, wall time, token usage, and malicious/safe fixture decisions.
The recall runner intentionally does not use skill-scanner 2.0.13 bundled
benchmark_runner.py: that runner constructs the core scanner without enabling the LLM or meta analyzers, so it cannot measure the configured model.Raw scan outputs and cloned sources remain gitignored because they are large and reproducible. The dated report preserves the durable comparison record.
Validation
scripts/skill-scan/eval/bench_recall.py --helpgit diff --checkRelated to #909.