Skip to content

skill: mandatory-ID gate for re-extracted docs on --update#2162

Open
AirRocker wants to merge 1 commit into
Graphify-Labs:v8from
AirRocker:feat/mandatory-id-gate
Open

skill: mandatory-ID gate for re-extracted docs on --update#2162
AirRocker wants to merge 1 commit into
Graphify-Labs:v8from
AirRocker:feat/mandatory-id-gate

Conversation

@AirRocker

Copy link
Copy Markdown

Problem

Semantic re-extraction of a doc that already has nodes in graph.json is non-deterministic. Running --update twice on the same large project CLAUDE.md — changed by one line between runs — returned 75 nodes on one run and 49 on the next, silently dropping an entire documented section.

Nothing errored. The loss went straight into build_merge and would only have surfaced much later as a question the graph could no longer answer. It was caught solely because the to_json shrink guard (#479) refused the write.

The failure mode is quiet by construction: the extraction "succeeds", the chunk is valid JSON, and the merge is happy. Only a node-count comparison against the prior build reveals it.

Fix

Adds a required three-step gate to the --update flow, applied to any changed doc/paper that already has nodes in the graph:

  1. Snapshot that file's existing node IDs from graph.json before extracting.
  2. Pass them to the subagent as mandatory must-include IDs, with the baseline node count as an explicit target ("the prior build extracted N nodes; land at ~N — under 90% means you under-extracted").
  3. Hard-gate the merge — refuse unless node count ≥ 90% of baseline and zero mandatory IDs are missing. On failure, re-dispatch naming the missing IDs; two consecutive failures stop and report rather than merging a regression.

It also restates that the shrink guard is the authoritative backstop: never force=True past it without diffing the old and new node sets and being able to name why each removed node is legitimately gone (file deleted, section removed, ID renamed with a verified replacement). ID churn on a re-extracted file is a legitimate shrink; a missing section is not.

Reusing the prior IDs has a second benefit beyond completeness: it suppresses gratuitous ID churn on re-extraction (claude_weather_proxy_workerclaude_workers_weather_proxy), which orphans saved queries and inflates the merge diff for no semantic gain.

Evidence

  • The 75 → 49 collapse above, caught only by the shrink guard.
  • After adopting the gate: three consecutive regression-free doc updates on the same project.
  • First run with the gate codified: 111/111 mandatory IDs present, node count held exactly, merge clean, health check clean.
  • A second, worse collapse (−26 nodes, dropping an entire documented API-contract section) was caught by the gate before merge and fixed by re-dispatching with the ID list — the graph never regressed.

Scope

Markdown only — no code or CLI changes.

  • Hand-edited: tools/skillgen/fragments/references/shared/update.md (the gate) and tools/skillgen/fragments/core/core.md (a pointer from the --update section).
  • Everything else in the diff is regenerated via python -m tools.skillgen + --bless.

All five skillgen guards pass locally: --check, --audit-coverage, --schema-singleton, --monolith-roundtrip, --always-on-roundtrip.

Note on CI

tests/test_labeling.py::test_label_communities_batches_when_over_batch_size may show red. It is pre-existing and unrelated to this PR — it reproduces on an unmodified checkout with:

pytest tests/test_skillgen.py tests/test_labeling.py

Cause: label_communities runs batches in a thread pool (graphify/llm.py, workers = max(1, min(max_concurrency, n_batches))), and for backend="gemini" concurrency stays > 1. The test appends to calls from inside the worker, so it records completion order but asserts dispatch order ([100, 100, 50]); under a different thread schedule it observes [100, 50, 100]. It passes when its file runs alone, which is why it usually goes unnoticed.

Happy to split that into its own issue/PR if useful — I left it untouched here to keep this diff to instructions only.

…hed doc

Semantic re-extraction of a doc that already has nodes in graph.json is
non-deterministic. On a large project CLAUDE.md, successive --update runs
of the same unchanged-except-one-line file returned 75 nodes and then 49,
silently dropping a whole documented section. Nothing errored: the loss
went into build_merge and would only have surfaced later as a question the
graph could no longer answer. It was caught solely because the to_json
shrink guard (Graphify-Labs#479) refused the write.

Adds a required three-step gate to the --update flow, in the shared
references/update.md fragment plus a pointer from the core template:

1. Snapshot that file's existing node IDs from graph.json before extracting.
2. Pass them to the extraction subagent as MANDATORY must-include IDs, with
   the baseline node count as an explicit target. Reusing the IDs also
   suppresses gratuitous ID churn, which orphans saved queries and inflates
   the merge diff for no semantic gain.
3. Hard-gate the merge: refuse unless node count >= 90% of baseline AND zero
   mandatory IDs are missing. On failure, re-dispatch naming the missing IDs;
   two consecutive failures stop and report rather than merging a regression.

Also restates that the shrink guard is the authoritative backstop: never
force past it without diffing the old and new node sets and being able to
name why each removed node is legitimately gone.

Fragments are the edited source; graphify/skill*.md, graphify/skills/**,
graphify/always_on/** and tools/skillgen/expected/** are regenerated via
`python -m tools.skillgen` and `--bless`. Markdown only, no code changes.

All five skillgen guards pass (--check, --audit-coverage, --schema-singleton,
--monolith-roundtrip, --always-on-roundtrip).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant