viz smart: bridge declared pipelines whose stages do not nest (#4222)

[jqnatividad]

Follow-on to #4281, from a design review of the NYC Capital Projects dashboard in the gallery.

The finding

That dashboard's headline panel drew a funnel that widens at the bottom. Spent totals 2.9× Committed, so plotly printed 285% on the last band and drew it wider than the one above it.

A funnel is a containment form — its band widths are the claim that each stage is a subset of the one before. Drawing one over stages that grow asserts the opposite of the data, and a subtitle cannot retract it, because the geometry is read before the caption is.

The dashboard already contained the refutation: its own Spearman panel reports ρ(Planned, Spent) = −0.11 and ρ(Planned, Committed) = 0.22. At the project level these three columns are near-unrelated — exactly what the source semantics say, since totalplannedcommit is allocated in the Capital Commitment Plan while the other two are sums within the City's budget. The hover row counts are non-monotonic too (73% → 40% → 55%). Neither dollars nor rows nest.

The fix: the declaration picks the panel, the numbers pick the form

The dictionary declaration was never the problem. It says these columns are one ordered pipeline — which no heuristic can infer, and which the tool should keep believing. What it cannot say is that the stages nest; that is measurable. Until now the declaration decided both whether to draw and what to draw. Those are separated:

• PipelineForm::for_totals draws a funnel only while the declared totals never grow.
• Otherwise the panel bridges the signed differences between consecutive totals — same declared order, same numbers, but the form asserts arithmetic rather than containment.

Deliberately not the per-row containment share, which measures something else and stays a subtitle disclosure: rows can overrun while the totals still shrink (a funnel remains truthful), or nest perfectly while the totals grow (it does not).

CPDB now reads Pipeline bridge: … · stages do not nest — bridged, not funnelled, stepping 192G → −163G → 28.4G → +52.6G → 81.0G. Each step is labelled as the arithmetic difference it is (Committed − Planned), never as a flow, since no flow between independent aggregates is claimed.

The bridge is a Waterfall trace, drawn vertically: a waterfall is an ordinary bar-like trace, so category index 0 lands at the axis bottom rather than the top, and the arrays cannot be reversed to compensate because it accumulates in array order and index 0 must seed the running total. Requires the Waterfall trace added in dathere/plotly#432; Cargo.lock moves to that rev (the branch = pin is unchanged).

The rule found a second real case beyond CPDB: a row-encoded marketing pipeline summing revenue grows down its stages even though the row counts shrink — revenue at conversion is not a subset of revenue at impression — so that bridges too.

Three labelling defects from the same review

• The completeness clause asserted the opposite of its purpose. {:.0} rendered 12,566-of-12,587 rows as 100% of rows, while $119,962,000 quietly went missing from the Planned bar. It now floors to one decimal, so only genuine completeness prints 100.
• One chart, two suffixes for 10⁹. Value axes carried plotly's untouched exponentformat: "B" default while qsv's own labels use d3 SI prefixes (%{y:.3s} on bars, .3~s on KPI indicators) — so a gridline read 35B and the bar against it 33.4G. The axes now declare SI, matching the convention qsv had already chosen rather than overturning it.
• A 416px margin that was masking a bug. The funnel panel borrowed the lollipop's budget for 56-char column-pair labels to hold ticks reading Planned. Sizing it to the real labels (87px) then exposed why that number had been survivable: plotly centres funnel bands on the value zero, so a 0..=ceil range puts every band's centre on the left edge of the plot area and draws half of it outside. The oversized margin had been absorbing the overflow. funnel_value_axis is now symmetric about zero.

Keeping the docs and the gallery honest

• USAGE and the generated docs/help/viz.md said a declared pipeline becomes a funnel. Half of that is still true — the declaration is the only source of the panel — but the form no longer follows from it.
• Routing CPDB to a bridge left PanelKind::Funnel with no gallery figure at all. That matters more than one less example: both funnel rendering bugs found so far were invisible in the serialized JSON and only surfaced in a browser. onboarding_funnel.csv (60 channel-weeks, fixed seed, nesting asserted at generation) restores that coverage, and renders through the typed subplot grid, which the inline renderer never exercises.
• gen_gallery.py now fails closed on README drift it can prove — the embedded-dashboard count, the screenshot link-out count, and whether every input fed to qsv is documented. It found an undocumented dataset immediately. Command text is deliberately not compared: the README is a tutorial whose invocations differ from the gallery's argv on purpose.
• cleanup_sidecars() now also runs before generation. viz smart reuses an existing stats cache rather than recomputing, so a cache left by unrelated --smarter work was silently inherited by a figure whose command never asked for it — which is how a committed dashboard ended up captioned signups (right-skewed, 3.3% outliers) beside a command that produces no such note.

Verification

259 viz unit + 354 test_viz integration tests pass (10 new/strengthened). Clippy clean apart from two pre-existing warnings. Verified in-browser at each step — the bridge, the restored funnel, and the axis/label changes — plus a static PNG export, which is what exercises the typed-grid arm, since the grid paths serve image export while HTML renders inline. Gallery regenerated with timestamp-only churn reverted, so the only artifacts that change are the ones whose figures actually did.

Four roborev rounds, each finding something smaller than the last, ending clean.

🤖 Generated with Claude Code

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}