fix: make checkpoint resume crash-consistent

[KAIWEILIUCC]

Summary

This PR makes SkillOpt-Lite checkpoint recovery crash-consistent.

Previously, resume logic primarily trusted runtime_state.json or the last
entry in history.json. A process interruption between checkpoint writes could
leave runtime_state.json, history.json, step records, and skill snapshots
out of sync. Partial step artifacts could also be reused during the next run.

What changed

• Jointly validate:
  • runtime_state.json
  • history.json
  • steps/step_XXXX/step_record.json
  • current and best skill snapshots
• Add commit IDs and skill-content hashes.
• Find the last contiguous, fully committed step N.
• Remove artifacts belonging to steps after N, including:
  • step directories
  • skill snapshots
  • ahead-of-checkpoint history entries
  • duplicate or ahead-of-checkpoint LR records
  • recognized branch, operation, and version-graph artifacts
  • incomplete slow-update and meta-skill artifacts
• Fully rerun step N + 1 instead of reusing its partial directory.
• Write checkpoint files atomically using:
  • a temporary file in the destination directory
  • file fsync
  • os.replace
  • directory fsync
• Write runtime_state.json last as the commit marker.
• Store immutable per-step best-skill snapshots.
• Restore trajectory digests for already committed steps in the current epoch.
• Add a run fingerprint to prevent resuming with a changed configuration or
  initial skill.
• Reject concurrent trainers using the same out_root.
• Atomically update and deduplicate lr_history.jsonl.

No new command-line arguments or configuration fields are required.

Recovery semantics

If a run is interrupted while executing step N + 1:

1. The next run validates all committed checkpoint artifacts.
2. It identifies step N as the last consistent commit.
3. It removes partial artifacts after step N.
4. It restores the committed skill and optimizer state.
5. It reruns step N + 1 from the beginning.

This prevents a partially written step from being treated as complete.

Tests

Added fault-injection coverage for:

• a consistent checkpoint;
• crashes at every checkpoint commit write boundary;
• runtime_state.json being ahead of other artifacts;
• mismatched runtime and step records;
• corrupted current-skill snapshots;
• corrupted baseline markers;
• cleanup of LR, branch, and version-graph artifacts;
• changed run fingerprints;
• concurrent writers targeting the same checkpoint directory.

Commands run:

python -m ruff check \
  skillopt/engine/__init__.py \
  skillopt/engine/trainer.py \
  skillopt/engine/checkpoint.py \
  tests/test_checkpoint.py

python -m unittest discover -s tests -v
python -m compileall -q skillopt tests
git -c core.whitespace=cr-at-eol diff --cached --check

[Copilot]

Pull request overview

This PR introduces crash-consistent checkpoint commit + recovery for the SkillOpt-Lite training loop, ensuring that partially written checkpoint artifacts are detected and rolled back so the next run resumes from the last fully committed step.

Changes:

• Added a dedicated crash-consistent checkpoint module with atomic file writes, step commit markers, recovery validation, and artifact pruning.
• Updated the trainer to use the new checkpoint commit/recovery flow, including run fingerprint validation and exclusive writer locking.
• Added fault-injection tests covering crash-at-boundary scenarios and recovery cleanup behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
tests/test_checkpoint.py	Adds recovery and crash-boundary fault-injection coverage for checkpoint consistency.
skillopt/engine/trainer.py	Switches checkpoint persistence/resume logic to the new crash-consistent checkpoint utilities and adds an out_root writer lock + fingerprinting.
skillopt/engine/checkpoint.py	New module implementing atomic writes, step commits, recovery validation, pruning, and locking utilities.
skillopt/engine/__init__.py	Lazily exposes ReflACTTrainer to avoid eager imports while keeping engine exports stable.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.