ulw-loop checkpoint quality gate schema and artifact-dir errors are hard to recover from

[yangwuan55]

Summary

omo ulw-loop checkpoint --quality-gate-json is difficult to recover from when the quality gate JSON shape or artifact location is wrong. The docs/source/runtime path give confusing signals about whether the JSON should be wrapped in a top-level qualityGate key, and the artifact-dir failure only becomes clear after several checkpoint attempts.

Environment

• LazyCodex / OMO version: 4.19.4
• Codex version: codex-cli 0.146.0
• OS: macOS / zsh
• Install method: Codex plugin cache under ~/.codex/plugins/cache/sisyphuslabs/omo/4.19.4
• Target workflow: $omo:ulw-loop final checkpoint --status complete with --quality-gate-json

Repository Decision

• Target repository: code-yeongyu/lazycodex
• Why this belongs there: the failing behavior is in the LazyCodex/OMO bundled ulw-loop component and skills, not upstream Codex CLI.
• LazyCodex evidence:
  • Current LazyCodex source checkout: code-yeongyu/lazycodex@fb48ddc.
  • plugins/omo/components/ulw-loop/src/quality-gate-fields.ts:13-15 emits generic section errors like Final quality gate is missing ${field} evidence.
  • plugins/omo/components/ulw-loop/src/quality-gate.ts:145-151 parses the gate through section(input, "qualityGate"), while the workflow docs show a bare object under --quality-gate-json.
  • plugins/omo/skills/ulw-loop/references/full-workflow.md:197-205 documents the --quality-gate-json shape as top-level codeReview, manualQa, gateReview, iteration, and criteriaCoverage without an outer qualityGate wrapper.
  • plugins/omo/components/ulw-loop/src/quality-gate.ts:93-105 enforces that all referenced artifacts are inside currentAttemptDir.
  • plugins/omo/components/ulw-loop/src/checkpoint.ts:230-235 passes currentAttemptDir into quality-gate validation for evidence layout v2.
  • plugins/omo/components/ulw-loop/src/codex-goal-instruction.ts:111 documents the current-attempt artifact requirement, but users only discover the exact required directory after checkpoint rejection unless they inspect generated instructions/source.
• Upstream Codex source evidence:
  • Current upstream source checkout: openai/codex@3aae5d8.
  • rg "ulw-loop|quality-gate-json|Final quality gate" returned no matches in upstream Codex, so ownership is not upstream Codex CLI.

Reproduction

This happened during a real final checkpoint close for a rougurlike $omo:ulw-loop task where all criteria were already recorded as pass.

1. Create/record all ULW criteria as pass for the active final story.
2. Prepare a quality gate JSON from the visible workflow docs, or wrap the same object while trying to infer the expected schema.
3. Run a checkpoint command of this form:

omo ulw-loop checkpoint \
  --goal-id G001-game-studio-sprite-pipeline-users-ym \
  --status complete \
  --evidence "all criteria C001-C003 PASS; codeReview/manualQa/gateReview PASS" \
  --codex-goal-json .omo/ulw-loop/evidence/cyberpunk-assets/final6/codex-goal-complete.json \
  --quality-gate-json .omo/ulw-loop/evidence/cyberpunk-assets/final6/quality-gate.json \
  --json

Observed failures during recovery included:

ULW_LOOP_QUALITY_GATE_INVALID: Final quality gate is missing codeReview evidence

and then, after adjusting the JSON shape:

ULW_LOOP_QUALITY_GATE_INVALID: manualQa.artifactRefs[0].path (.omo/ulw-loop/evidence/cyberpunk-assets/final6/05-battle.png) must point to an artifact from the current attempt (.omo/evidence/ulw/019fdfed-9dd0-7c30-bb65-9c6d7775c4fc/G001-game-studio-sprite-pipeline-users-ym/a0).

The second error is technically correct, but the workflow made it easy to build valid-looking quality-gate evidence in the wrong directory and only discover that at final checkpoint time.

Expected Behavior

The checkpoint path should make quality-gate recovery straightforward:

1. The accepted JSON shape should be unambiguous and consistent across docs, source, and runtime.
2. If the user passes { "qualityGate": { ... } } when the runtime expects a bare object, or a bare object when the runtime expects a wrapper, the error should explicitly say which form is accepted.
3. omo ulw-loop status --json or checkpoint preflight output should expose the exact currentAttemptDir that quality-gate artifact paths must live under.
4. The checkpoint error should suggest the next action, e.g. copy/move artifacts under the current attempt directory or regenerate the quality gate there.

Actual Behavior

• A schema mismatch can surface as Final quality gate is missing codeReview evidence, which points at the missing nested section rather than the real issue: the top-level JSON shape is not what the runtime expected.
• The docs and source are confusing to reconcile:
  • docs show a bare quality-gate object as the --quality-gate-json shape;
  • current source parses section(input, "qualityGate") before reading codeReview.
• Artifact path validation is strict and correct, but users may only learn the exact required currentAttemptDir after a failed final checkpoint.

Evidence

Runtime/environment evidence:

omo --version => 4.19.4
codex --version => codex-cli 0.146.0

Source evidence from code-yeongyu/lazycodex@fb48ddc:

plugins/omo/components/ulw-loop/src/quality-gate-fields.ts:13-15
section(value, field) => invalid(`Final quality gate is missing ${field} evidence.`, field)

plugins/omo/components/ulw-loop/src/quality-gate.ts:145-151
validateQualityGate(input) reads section(input, "qualityGate") before codeReview/manualQa/gateReview/iteration/criteriaCoverage.

plugins/omo/skills/ulw-loop/references/full-workflow.md:197-205
The documented --quality-gate-json example is a bare object with top-level codeReview/manualQa/gateReview/iteration/criteriaCoverage.

plugins/omo/components/ulw-loop/src/quality-gate.ts:93-105
checkFile rejects artifacts outside currentAttemptDir.

plugins/omo/components/ulw-loop/src/checkpoint.ts:230-235
checkpoint passes currentAttemptDir into validateQualityGate for evidence layout v2.

Upstream routing evidence:

openai/codex@3aae5d8
rg "ulw-loop|quality-gate-json|Final quality gate" => no matches

Root Cause

There appear to be two related UX/contract issues:

1. The --quality-gate-json schema contract is not consistently communicated. The source path currently suggests a top-level qualityGate wrapper, while the workflow docs show the bare gate object. In practice, a mismatch leads to a generic missing-section error instead of a shape-specific correction.
2. Evidence layout v2 correctly enforces currentAttemptDir, but that required directory is not surfaced early enough in the user-facing recovery path. Users can collect real evidence under another directory, build a complete gate, and only then discover that the artifact refs are invalid for checkpoint completion.

Proposed Fix

• Align the docs, source, and runtime on one accepted --quality-gate-json shape. If backward compatibility is desired, accept both:

const gate = isRecord(input) && isRecord(input["qualityGate"])
  ? input["qualityGate"]
  : input;

• Add targeted validation for wrapper mistakes:
  • if the runtime expects a bare object and receives { qualityGate: ... }, say: --quality-gate-json expects the gate object directly; remove the outer qualityGate key.
  • if the runtime expects a wrapper and receives top-level codeReview, say: --quality-gate-json expects { "qualityGate": ... }; wrap the gate object.
• Include currentAttemptDir directly in omo ulw-loop status --json for the active goal, or add a quality-gate preflight/checkpoint --dry-run command that prints it.
• In the artifact path error, include a suggested recovery step such as:

Copy or regenerate quality-gate artifacts under:
.omo/evidence/ulw/<session>/<goalId>/a<attempt>/

• Add regression tests covering:
  • wrapped vs bare quality gate JSON;
  • documented sample shape;
  • artifact refs outside currentAttemptDir with a recovery hint;
  • status --json exposing the attempt dir used by checkpoint validation.

Verification Plan

• Run the existing ulw-loop quality gate tests.
• Add a failing-first test using the documented quality-gate sample from skills/ulw-loop/references/full-workflow.md.
• Add a checkpoint validation test where artifact refs are outside the current attempt directory and assert that the error includes the actual currentAttemptDir plus recovery guidance.
• Run an end-to-end omo ulw-loop checkpoint --quality-gate-json <sample> flow and confirm the user-facing error points directly to the schema or artifact-location issue.

---

This issue or PR was generated by LazyCodex.
Tag: lazycodex-generated