[skill-optimizer] Daily Skill Optimizer Improvements - 2026-04-24

[github-actions]

Summary

• Run mode: dry-run
• Status: ❌ Run failed (run_status: 1) — optimizer could not execute due to uncommitted changes in the workspace

Key Findings

The skill-optimizer dry-run was blocked before it could evaluate any prompts. The root cause is the optimize.requireCleanGit guard, which enforces a clean git working tree before optimization can proceed. Since the CI workspace had uncommitted changes at run time, the process exited with an unhandled error. This means no prompt quality data was collected, but the failure itself reveals three concrete, actionable improvements:

1. Ensure a clean git state before the optimizer job runs — add a git stash or explicit checkout step in the CI workflow so the optimizer never sees dirty state.
2. Surface the requireCleanGit failure as a workflow annotation, not a silent artifact failure — the error is buried in run.log; a step-level failure annotation would make it immediately visible in the Actions UI.
3. Add a smoke-test task to SKILL.md that covers the optimizer's dry-run path, so future regressions (blocked runs, config drift) are caught by the task suite rather than discovered through failed nightly runs.

Recommendations

1. Fix CI setup: stash or reset workspace before optimizer runs

   • The optimizer requires a clean git working tree (optimize.requireCleanGit: true).
   • In .github/workflows/ (the workflow that invokes the skill-optimizer job), add a step git stash --include-untracked (or git checkout -- .) immediately before the skill-optimizer optimize command.
   • Expected impact: Eliminates the run_status: 1 failure class entirely; the optimizer will proceed to evaluate prompts on every scheduled run.

2. Emit a clear GitHub Actions step summary / error annotation on run_status != 0

   • Currently, failures are only visible by downloading and reading run.log. The Actions UI shows the job as "succeeded" at the workflow level because the step exits 0 after uploading the artifact.
   • Add logic (or a post-step) that checks summary.json's run_status and calls echo "::error::skill-optimizer run failed — see run.log" when non-zero.
   • Expected impact: On-call engineers and PR authors see the failure immediately in the Actions UI without manually downloading artifacts, reducing mean-time-to-detect for optimizer regressions.

3. Add a minimal dry-run validation task in SKILL.md

   • SKILL.md currently does not include any task that exercises the optimizer's own setup path (config loading, git-clean check, model list validation).
   • Add a task such as: "Run skill-optimizer dry-run --config .skill-optimizer/skill-optimizer.json from a clean checkout and confirm it exits 0 with the expected model and task-count summary printed."
   • Expected impact: Future config drift (e.g., a model name change, a schema bump) will be caught in the task suite before it silently breaks nightly optimization, giving faster feedback loops.

Evidence from Artifact

summary.json

{
  "repository": "github/gh-aw",
  "run_mode": "dry-run",
  "init_status": 0,
  "run_status": 1,
  "run_url": "https://github.com/github/gh-aw/actions/runs/24916318508"
}

run.log (tail)

Unhandled error: Error: target.repoPath: target repo has uncommitted changes
  (optimize.requireCleanGit is enabled) — Run: git stash or commit your
  changes in /home/runner/work/gh-aw/gh-aw
    at validateProjectConfig (.../project/validate.js:425:15)
    at async loadProjectConfig (.../project/load.js:30:5)
    at async runDryRun (.../cli.js:191:21)
    at async main (.../cli.js:222:9)

init.log (status 0 — init succeeded)

[init] Updated /home/runner/work/gh-aw/gh-aw/.skill-optimizer/skill-optimizer.json
[init] Done!
  Surface:    prompt
  Models:     1 — claude-sonnet-4.6
  Tasks:      up to 20 per run
  Iterations: up to 3
  Target:     80% pass rate

The init phase completed successfully, confirming the config is valid. The failure is isolated to the run phase's git-state pre-check.

Generated by Daily Skill Optimizer Improvements · ● 209.2K · ◷

• expires on May 1, 2026, 11:26 PM UTC

[pelikhan]

@copilot disable requireCleanGit