viz smart: dictionary-declared pipeline funnel (#4222)

[jqnatividad]

Closes #4222 — the last of its three asks. Asks 1 and 2 (the Lorenz inequality caveat) shipped in #4280.

This PR was substantially redesigned mid-flight. It began as a column-name vocabulary plus a row-wise containment gate. It now derives the funnel from an explicit data-dictionary declaration, with containment kept as a disclosure rather than a gate. The reasoning is below, because the earlier design's own headline finding is what argued against it.

Problem

The pipeline in the NYC Capital Projects data was visible only as % zeros annotations scattered across three separate box titles: 47% of projects have nothing committed, 55% nothing spent. Read one at a time those are three unrelated data-quality notes. Read together they are a funding pipeline — the actionable governance signal. The funnel states it once.

Why the first design was wrong

Issue #4222 framed the question correctly for its sibling asks:

how does viz smart recognise that a set of measures forms a pipeline? … This has the same shape as the unit-heterogeneity problem above — it is semantics, not a statistic — so it deserves the same explicit treatment rather than a guessed proxy.

Asks 1 and 2 were resolved exactly that way: no gate, an unconditional caveat, no false negatives. Ask 3 initially shipped with the guessed proxy that reasoning rejects — a hardcoded word list to guess with (FUNNEL_FAMILIES: budget / marketing / fulfillment), and row-wise containment as a gate to catch its own bad guesses.

The evidence against it came from the motivating dataset itself. Real CPDB is not row-wise nested — spent_total is 2.9× commit_total, violating containment on 46.3% of rows — because the three columns are independent aggregates on different accounting bases, which the dataset's own column descriptions explain. The old design's response was to draw nothing at all. But conceptually those three columns are planned → committed → spent. Refusing to chart them was a false negative produced by a statistical proxy standing in for a semantic question.

What it does now

A funnel is drawn only when the dictionary declares one, via a kind: "pipeline" entry in x-qsv.relationships. Nothing is guessed.

The enabling discovery: describegpt already inferred this and viz never saw it. The dictionary prompt has always asked for a top-level relationships array, and parse_llm_relationships has always validated it — but format_dictionary_jsonschema dropped it on the floor, and JSON Schema is the format viz smart --dictionary consumes. Relationship inference is gated on --infer-content-type, which --dictionary infer already passes. So the LLM was being asked for this on exactly viz's code path, and the answer was discarded at emission.

Two encodings, both hand-editable in the saved <stem>.schema.json:

// stages as COLUMNS — upstream first. NB: the OPPOSITE direction from "ordered", which ascends.
{"kind":"pipeline","members":["totalplannedcommit","commit_total","spent_total"]}

// stages as ROW VALUES of one category column — the shape smart previously declined to guess at
{"kind":"pipeline","members":["stage","revenue"],"stage_column":"stage",
 "stages":["Impression","Click","Lead","Conversion"],"value_column":"revenue"}

Containment is measured, never enforced. A declared pipeline whose stages don't nest is drawn with the violation share named in the subtitle. CPDB now renders for the first time, reading "Spent exceeds Committed in 46% of rows · Spent total exceeds Committed — overruns, not leakage" — the same 46% the old design used to refuse on. More informative than silence.

Declared order is authoritative and is never re-sorted by size, so a stage that outruns its predecessor stays visible instead of being quietly reordered away.

The one guard that survived, and why

FUNNEL_COMPLEMENT_MARKERS (unspent, remaining, balance, …) outlived the vocabulary it was born in, repurposed from a veto into a warning.

This is the single bad declaration that disclosure cannot catch. A complement column nests perfectly inside its predecessor — it reports 0% violations — while rendering a funnel that means the opposite of the truth. Every other bad declaration shows up as a violation share the subtitle can name; this one does not.

The old list's collision half (review, contractor, wholesale, lead_time, replaced) was deleted with the vocabulary. Those words existed solely as antidotes to greedy substring matching; warning on a column a user explicitly declared would be a pure false alarm.

Deliberate cost: a dedicated data pass

The funnel no longer piggybacks the correlation read. That read excludes near-unique columns, so a declared money column could silently vanish from it — and force-including one would perturb the Pearson matrix and shrink the listwise join for every other correlation-derived panel.

It also fixes a live bug: the complete-case denominator was previously diluted by columns unrelated to the funnel. It now covers exactly the declared stages, and a test pins it.

Plotly dependency

Unchanged. Uses the Funnel trace added in dathere/plotly (upstream PR plotly/plotly.rs#432); Cargo.toml:475 pins the stacked deps/webdriver-downloader-0.17+funnel branch, so qsv is not blocked on the upstream merge.

Funnel is a cartesian trace, so unlike Sankey it composes with the typed subplot grid and static export — verified by rendering to PNG, not assumed. Gotcha for future readers: a funnel draws index 0 at the top, the opposite of a plain category axis.

Verification

• 255 viz unit + 352 test_viz + 179 describegpt unit + 86 describegpt integration + 39 synthesize. Clippy clean (2 pre-existing warnings).
• End-to-end on the motivating dataset: CPDB draws, subtitle reports 46% — matching the share recorded on the issue, so the measurement is unchanged and only the gate is gone.
• Dictionary-only confirmed: all 20 gallery corpus CSVs produce zero funnels without a dictionary, including the perfectly-nested fixture that used to produce one.
• jsonschema::meta::validate passes with relationships in root x-qsv — asserted in a test, not assumed.

The vacuous-test trap, handled explicitly

Under dictionary-only, the four old !contains("Pipeline funnel:") negatives all still pass — for an entirely different reason than they were written for (no dictionary, no funnel). They were green while testing nothing. They are replaced by one honest baseline, viz_smart_no_funnel_without_a_dictionary, which asserts that the nested table that does draw with a dictionary draws nothing without one — and which would have failed against the old code, since the vocabulary charted that exact table.

A cross-command invariant is also pinned: SynthRelationship::members has no serde default (unlike anchor) and the dictionary deserializes in one pass, so a pipeline entry lacking members would hard-error synthesize --dictionary rather than degrade. members is therefore synthesized for the row encoding rather than taken from the LLM.

Known limits

• The LLM can mis-declare a partition as a pipeline — a status column on a one-row-per-entity table. The prompt guards against it explicitly ("check the grain sentence you are about to write"), but the guard depends on the model obeying. Editing the saved schema is the fix, which is why the declaration lives in a hand-editable file.
• Row-encoded funnels use the disjoint reading (counts are rows at each stage). A cumulative pipeline — one row per entity, column = furthest stage reached — would need a reverse-cumulative sum that is only valid under an assumption the data cannot confirm. A future stage_semantics key can lift this.
• Legacy flat {"fields":[…]} dictionaries are unsupported for pipelines (JSON Schema only); they have nowhere to put a row-encoded declaration.
• viz smart on a plain CSV will never show a funnel again. That is the intended trade: no false positives, at the cost of requiring a dictionary.

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 6 complexity

Metric	Results
Complexity	6

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}