Skip to content

feat(cli): add offline result compare command #175

Description

@googs1025

Background

Now skill-up can run evals, generate reports, and create baseline benchmark data. But I think it still misses one offline command to compare two finished runs.

When we change a Skill, review a PR, or upgrade model, we often want to know:

  • which cases are fixed
  • which cases are regressed
  • how pass rate changes
  • how tokens and duration change
  • whether engine or model is different
  • whether this can be used as a CI gate

Proposal

Add a new command:

skill-up compare <old-result.json> <new-result.json>

It should compare two existing result/report JSON files. It should not run eval again, and should not call any agent or judge.

Scope

The command can:

  • read JSON files compatible with current report.Input/result.json format
  • use primary case result semantics, so benchmark without_skill entries are not counted twice
  • show run level delta:
    • pass rate
    • case counts
    • total/input/output tokens
    • duration
  • show case level transition:
    • fixed
    • regressed
    • unchanged
    • added
    • removed
  • show metadata difference, like skill name, engine, model, schema version and time
  • support default text output
  • support --format json for CI or other tools
  • support optional gates, for example fail on regression or too much token increase

Non-goals

  • Do not rerun evals
  • Do not call agent or judge
  • Do not add dashboard/cloud/snapshot feature in this issue
  • First version does not need markdown output

Acceptance criteria

  • skill-up compare old.json new.json prints a readable summary
  • --format json prints stable machine readable output
  • benchmark result is not double counted
  • fixed/regressed/added/removed cases are classified correctly
  • configured gate failure returns non-zero exit code
  • invalid input gives clear error message

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status
    In progress

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions