From d5b115f9d48c963b73d43ff97d1252c5c4282acd Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 16 Aug 2026 14:27:53 +0000 Subject: [PATCH 1/3] spec(LTX25-DECODE-DTYPE): the decode's accumulator is f64, and torch's is f32 even in bf16 (#1008) Lever 2 of the LTX25-DECODE-SPEED investigation (#1006, PR #1018) split into the dtype half. The parent row filed #1008 on the observation that the LTX-2.5 conv video VAE decode declares `double acc` at 8 sites and casts 29 operands to double, where no reference stores f64 anywhere on the path. Reading the references gives the tensor dtype, not the accumulator width, so this spec settles the width by running torch 2.11.0 on a reduction engineered to separate the two: 27 taps over a uniform input with weights [+1e8, 0.1 x 25, -1e8], where every partial sum sits below half an ulp of 1e8 and an f32 accumulator lands on exactly zero in any summation order. F.conv3d returns 0.0 on f32 tensors AND on bf16 tensors, upstream's own configuration; the f64 arm returns 2.5, which is what this port returns today. That measurement is what makes the new gate non-circular: the value it asserts is torch's answer, not a recording of the patched code. It also bounds what may be gated, and the spec records the bound rather than overselling it -- torch.sum and torch.mean on the same input return 2.0999999, so torch's reductions are f32-wide but cascading where this port is naive serial. Width matches, order does not, so no adversarial gate is built through PixelNorm. The spec also records why the goldens cannot see any of this, from their own generator rather than by inheriting the claim: gen-ltx2-vae-goldens.py:223 casts every upstream parameter to np.float32, so the oracle that produced them ran f32 end to end. The useful corollary is that this port has been accumulating WIDER than the oracle its own goldens came from. The NDHWC half of the lever is filed, not half-built, and the spec names the blocker precisely: it is not `Volume`, whose At() has 16 call sites, but MiniMaxH3GroupNorm3d, whose signature hard-codes a channel-major buffer and which MiniMax-H3's VAE CNN and the LTX-2 audio VAE also call. No speed number is claimed. dgx.casa was unreachable throughout, so the magnitude is booked as owed and unmeasured with what would settle it. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/ltx25-decode-dtype.md | 264 ++++++++++++++++++++++++++++ 1 file changed, 264 insertions(+) create mode 100644 .agents/specs/ltx25-decode-dtype.md diff --git a/.agents/specs/ltx25-decode-dtype.md b/.agents/specs/ltx25-decode-dtype.md new file mode 100644 index 000000000..c09865fab --- /dev/null +++ b/.agents/specs/ltx25-decode-dtype.md @@ -0,0 +1,264 @@ +# LTX25-DECODE-DTYPE — the decode computes in f64, and no reference does + +Row: `LTX25-DECODE-DTYPE`, under the `ROAD-V1-LTX25` campaign +([`roadmap_v1.md`](../roadmap_v1.md), [`ltx-2-5.md`](ltx-2-5.md)). +Issue: [#1008](https://github.com/mudler/vllm.cpp/issues/1008). +Parent: lever 2 of the `LTX25-DECODE-SPEED` investigation +([#1006](https://github.com/mudler/vllm.cpp/issues/1006)), which filed this +issue and lists it under `## Owed`. That spec is +`.agents/specs/ltx25-decode-speed.md` on [PR +#1018](https://github.com/mudler/vllm.cpp/pull/1018) and is **not yet on +`main`**, so it is cited by pull request rather than by relative link — a +relative link would dangle until #1018 lands. + +## Now + +`ACTIVE`. This row takes the *dtype* half of lever 2. It does not take the +memory-format half; §5 states the verdict and what blocks it. + +## 0. Scope, and the one thing this row is not + +**In scope.** Every accumulation and every elementwise arithmetic step on the +LTX-2.5 conv video VAE data path moves from `double` to `float`, which is the +dtype upstream actually computes these ops in. Scalar constants — epsilons, +`1/sqrt(C)`, per-channel shift/scale precomputed from two f32 weights — stay +`double`, because upstream's counterparts are Python floats and they cost +nothing. + +**Not in scope, and deliberately so.** The *storage* dtype stays f32. Upstream +stores bf16 (§1), and moving this file to bf16 storage is the production arm +that `ltx2_video_vae.cpp:46-49` already books as phase L6 debt. That is a +different change with a different risk profile, and this row does not take it. +Narrowing the *arithmetic* is separable from narrowing the *storage*, and only +the first can be done without a second dtype for every buffer. + +**Also not in scope:** threading ([#1009](https://github.com/mudler/vllm.cpp/issues/1009)), +the device arm ([#1007](https://github.com/mudler/vllm.cpp/issues/1007)), +`memory_efficient_decode.py` ([#1011](https://github.com/mudler/vllm.cpp/issues/1011)). + +**No speed number.** `dgx.casa` has been unreachable for the whole of this row's +work, so there is no GPU and no large-render host. This row lands the +correctness-preserving change with the numerics pinned and books the magnitude +as owed and unmeasured (§7). An honest "unmeasured" beats an invented number, +and this campaign's own standing rule is that a number quoted often becomes +treated as measured. + +## 1. What dtype upstream actually accumulates in + +The issue and the parent spec both establish that no reference *stores* f64 on +this path. That is true and it is not the question this row had to answer. The +question is what dtype the *accumulation* happens in, which is not the same as +the tensor dtype, and it was settled by running the reference rather than by +reading it. + +**Read.** Every conv in the decoder is a plain `torch.nn.Conv3d` — `CausalConv3d` +builds it at `packages/ltx-core/src/ltx_core/model/video_vae/convolution.py:292-302` +and the single call site for the whole decoder is `:312`, at +`fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca`. `PixelNorm.forward` is +`torch.mean(x**2, dim=...)` in the activation dtype +(`model/common/normalization.py:37-40`). `_RMSNorm2D.forward` is +`F.normalize(x, dim=1) * (scale * gamma)`, and `AttnBlock3D.forward` is +`to_qkv` / SDPA / `proj`, all in the activation dtype +(`model/video_vae/attention.py:23, 58-69`). `diffusers` at `3a2f35d4e` agrees: +`PerChannelRMSNorm.forward` computes in the activation dtype +(`src/diffusers/models/autoencoders/autoencoder_kl_ltx2.py:50-59`). + +**Run.** Reading source gives the tensor dtype, not the accumulator width. The +accumulator was measured directly with torch 2.11.0, on a reduction engineered +so the two widths are separable: 27 taps over a uniform-1.0 input with weights +`[+1e8, 0.1 x 25, -1e8]`. Every partial sum `1e8 + j*0.1` for `j <= 25` is below +half an ulp of `1e8` (which is 4.0), so an f32 accumulator holds exactly `1e8` +until the final `-1e8` and lands on exactly zero, in **any** summation order. An +f64 accumulator lands on 2.5. + +| what | result | +|---|---| +| `F.conv3d`, f32 tensors | **0.0** | +| `F.conv3d`, bf16 tensors — upstream's own configuration | **0.0** | +| `F.conv3d`, f64 tensors | 2.500000014901161 | +| naive serial f32, by hand | 0.0 | +| this port today | 2.5 | + +**So upstream's convolution accumulator is f32, and this port's is f64.** The +expected value this row's new gate asserts is therefore torch's own answer, not +a recording of what the patched code happens to emit. + +**One honest caveat, recorded because it bounds what may be gated.** The same +probe run through `torch.sum` / `torch.mean` on f32 returns `2.0999999046325684`, +not `0.0`. Torch's reductions are f32-*wide* but use a cascading/pairwise order, +where this port's are naive serial. Width matches; order does not. A gate that +pinned an adversarial value through `PixelNorm` would therefore be pinning this +port's summation order rather than upstream's dtype, so this row does not build +one. The convolution is the site where torch's answer and a naive serial f32 +answer coincide exactly, and it is also where 42 of the decode's convolutions +and essentially all of its arithmetic live. + +## 2. The sites, and what each one becomes + +The issue counts 8 `double acc` declarations and 29 `static_cast`. The +`acc` count is right and is not the whole set: `PixelNorm`'s `mean_sq`, the +attention block's `sum_sq`, `dot` and `sum`, and the attention block's six +`std::vector` activation scratch buffers are accumulators and data-path +buffers that the `double acc` grep does not reach. + +Anchors are at `ff264cb82`, this row's base. + +| site | what it is | upstream | this row | +|---|---|---|---| +| `:165` | `CausalConv3d` output accumulator — 42 convs, ~all the FLOPs | `nn.Conv3d`, `convolution.py:292-302,312`; measured f32 (§1) | `float` | +| `:201` | `Linear3d`, the 1x1x1 conv used as `conv_shortcut` | `make_linear_nd` dims=3 -> `nn.Conv3d`, `convolution.py:84-85` | `float` | +| `:221` | `PixelNorm` `mean_sq` | `torch.mean(x**2)` in activation dtype, `normalization.py:37-40` | `float` | +| `:303`, `:312` | `TimestepEmbedding`'s two `nn.Linear` accumulators | `nn.Linear` in activation dtype | `float` | +| `:528` | attention `_RMSNorm2D` `sum_sq` | `F.normalize(x, dim=1)`, `attention.py:23` | `float` | +| `:546` | attention `to_qkv` accumulator | 1x1 `nn.Conv2d`, `attention.py:55` | `float` | +| `:557`, `:564` | attention score `dot` and softmax `sum` | SDPA in activation dtype, `attention.py:65` | `float` | +| `:570` | attention value-weighted accumulator | SDPA | `float` | +| `:579` | attention `proj` accumulator | 1x1 `nn.Conv2d`, `attention.py:56` | `float` | +| `:916` | encoder `SpaceToDepthDownsample` group mean | `.mean(...)` in activation dtype | `float` | + +Elementwise data-path steps narrow with them, for the same reason and the same +anchor: `Silu` (`:213`, `F.silu`), the ada-LN apply (`:361-362`), the noise +blend and per-channel denormalize (`:629-631`, `:645-648`), +`FeedSpatialNoise` (`:337-338`), and the encoder's normalize (`:1129-1130`). + +**What stays `double`, each with its reason written beside it in the file.** + +* Stabilizing epsilons (`norm_eps`, `pixel_norm_eps`, `kLtx2RmsNorm2dEps`) — + upstream's are Python floats, they are compared and added once per row, and + narrowing them changes a threshold rather than a data path. +* `norm_scale` = `sqrt(C)` and `attn_scale` = `1/sqrt(C)` (`:514-515`) — + upstream's `channels**0.5` is a Python float, evaluated once per block. +* The `TimestepEmbedding` frequency table (`:284-292`) — a transcendental + constant precompute, not a data-path accumulation, evaluated once per block + over 256 entries. Upstream's `torch.arange`-based table is f32; keeping f64 + here is *closer* to the value that table approximates, and it is not on any + hot path. It is annotated as a deliberate exception rather than left silent. + +## 3. The gate the goldens cannot be + +`ltx2_video_vae.cpp:41-44` already says why, and this row confirmed it rather +than inheriting it: `scripts/gen-ltx2-vae-goldens.py:223` casts every upstream +parameter with `values.astype(np.float32)`, and the input builders at `:759` and +`:836` do the same. **The oracle that produced the goldens ran f32 end to end.** +A dtype comparison against it cannot see an accumulator that is too wide, which +is the defect this row removes. + +That cuts both ways, and the second way is the useful one. Because the golden +generator ran torch in f32, the goldens are the output of an **f32-accumulating** +reference. This port's f64 accumulation has been *wider than the oracle its own +goldens came from* since the file landed. Narrowing to f32 moves this port +toward the goldens' generator, not away from it — which is why §6 expects the +recorded `max|diff|` to hold or improve, and treats a regression as a finding. + +**The new gate.** `test_ltx2_vae.cpp` gains one case that enters through the +production entry point `Ltx2VideoDecodeStreaming` and asserts the decode's +convolution accumulator is f32-wide, using §1's separable reduction. The +decoder is configured so the assertion is analytically derivable end to end +rather than recorded: + +* `decoder_blocks` empty, `timestep_conditioning=false`, `norm_layer=pixel_norm`, + `patch_size=1`, `out_channels=1`, `base_channels=2`, `in_channels=1`, + spatial padding `replicate`, latent all `1.0`, `std-of-means=1`, + `mean-of-means=0`. Every conv tap therefore sees exactly `1.0`, including at + every border, so one reduction is repeated at every output voxel. +* `conv_in` channel 0 carries `[+1e8, 0.1 x 25, -1e8]` with bias 0; channel 1 + carries all-zero weights with bias 1. +* f32 accumulator: ch0 = 0, ch1 = 1 -> `PixelNorm` leaves ch0 at 0 -> `SiLU(0)=0` + -> `conv_out` selects ch0 -> **every output element is exactly 0**. +* f64 accumulator: ch0 = 2.5, ch1 = 1 -> `PixelNorm` inv = `1/sqrt(3.625)` -> + ch0 = 1.3131 -> `SiLU` -> ~1.0348. The two arms are three decimal orders apart. + +The value `0` is upstream's measured answer for this reduction in both f32 and +bf16 (§1), so the case gates a mirrored property and not a local convention. + +## 4. Risks + +* **The narrowing moves a golden past `kLtx2GoldenTol` = 5e-6.** This is the one + risk that can change the design. Naive serial f32 is a worse summation order + than torch's cascading f32, so the error against the goldens can rise even + though the width now matches. Mitigation: measure `max|diff|` per golden arm + before and after, and report both. If an arm exceeds tolerance, the finding is + reported and the site is either kept `double` with its reason written beside it + or given a better summation order — never a widened tolerance. Widening a + tolerance to admit a change is the failure mode `AGENTS.md` forbids. +* **A `float` accumulator silently re-promoted.** `acc += float * double` promotes + the whole expression back to `double`. Every narrowed site must have its scalar + operands narrowed too, or the change is a no-op that reads as done. The §3 gate + catches this for the convolution; the reviewer should mutate the others. +* **`std::exp` overload selection.** `std::exp(-v)` with `float v` selects the + float overload; `std::exp(-static_cast(v))` does not. `Silu` depends on + this. + +## 5. NDHWC — the verdict, and why this row does not build it + +**The engine can express the layout. The blocker is not `Volume`, and naming it +as `Volume` would be wrong.** + +`Volume` is a file-local struct in an anonymous namespace +(`ltx2_video_vae.cpp:68-76`) whose `At()` is a single indexing function with 16 +call sites. That much is cheap. What is not cheap: + +1. **A shared helper hard-codes NCDHW in its signature.** + `MiniMaxH3GroupNorm3d(std::vector& x, int64_t channels, int64_t spatial, ...)` + (`include/vllm/model_executor/models/minimax_h3.h:756`) takes a + channel-major buffer by contract. The LTX-2.5 video VAE calls it at four + sites, and it is **shared**: MiniMax-H3's own VAE CNN calls it at three + (`minimax_h3_vae_cnn.cpp:176,182,337`) and the LTX-2 audio VAE at one + (`ltx2_audio_vae.cpp:215`). NDHWC in the video VAE means either an NDHWC + entry point on that shared helper, or a transpose at every norm boundary + which spends what the layout was meant to save. This is a shared-seam + decision, not a local one. +2. **Ten layout-dependent sites bypass `At()`** with open-coded `c * n + i` + arithmetic — `Linear3d` (`:203,:206`), `ApplyAdaLn` (`:361-362`), the + denormalize (`:645-648`), `SpaceToDepthDownsample` (`:919,:921`), the + encoder normalize (`:1129-1130`) — plus the local scratch buffers `padded`, + `normed`, `q`, `k`, `v` and `attended`, which carry their own layouts and + would each need one. +3. **`Ltx2VideoFrames::data` must not move.** It is the decode's output contract + and the frame writer's input. + +**And the win would not be the layout.** Upstream's `channels_last_3d` +(`memory_efficient_decode.py:617-627`, `:655-656`) exists to select a different +cuDNN 3-D convolution kernel family on a GPU. This port has no device arm at all +([#1007](https://github.com/mudler/vllm.cpp/issues/1007)), so on the host arm +that exists today NDHWC buys nothing by itself: it makes the reduction over +input channels contiguous, which is worth having **only once something +vectorizes over it**. The layout is a precondition for a SIMD or device arm, not +a speedup on a scalar loop nest. + +**Verdict: real, medium-sized, and a separate row.** It is filed rather than +half-built, and it must land after or with the arm that consumes it. The parent +spec's §8 says why bundling it here would be wrong: a dtype change no golden can +see and a memory-format change across a shared seam are two independent reviews. + +## 6. Gates and evidence + +1. **The new width case is RED before the change and GREEN after**, captured + with its exit code, not with a grep of the assertion line. +2. **Reachability.** The case enters through `Ltx2VideoDecodeStreaming`, the + entry `src/vllm/multimodal/ltx2_video.cpp:3258` calls on the render path. + Deleting the `Ltx2ConvVideoDecode` call at + `ltx2_video_vae_tiled.cpp:113` must turn it RED. +3. **`max|diff|` recorded for every video golden arm, before and after.** The + goldens must stay green on their own tolerance, and the two numbers are + reported side by side rather than summarized as "still passes". +4. **Full gate**: configure, build, `ctest`, with `: error:` count, test count, + exit codes, load and free disk. +5. Every mutation reports three facts: `git diff --stat` after applying, whether + it BUILT with the compile-error count, and the exit code. + +## 7. Owed + +| owed | what would settle it | +|---|---| +| **The speed magnitude of this change. UNMEASURED.** No number is claimed. | One `Ltx2ConvVideoDecode` wall at a fixed size on an idle host, same binary, f64 arm against f32 arm. `dgx.casa` was unreachable for this row's whole duration; any host that can run the decode settles it, and #1010's phase timings make it readable from a render. | +| NDHWC / `channels_last_3d` (§5) | its own row, after or with a SIMD or device arm | +| bf16 *storage*, the phase L6 production arm (`ltx2_video_vae.cpp:46-49`) | [#1007](https://github.com/mudler/vllm.cpp/issues/1007) and the L6 row | +| The `.agents/issue-index.md` row for [#1008](https://github.com/mudler/vllm.cpp/issues/1008) | It is **deliberately not appended here.** The row exists on PR #1018, which filed the issue and is unmerged. `.gitattributes:7` sets `merge=union` on that file and `scripts/check-agent-record.py:1437-1442` refuses a duplicate issue number with "duplicate is what two branches appending the same issue look like". Appending it here would turn `main` red for every branch the moment #1018 merges — the exact failure a duplicate #995 row caused. The link lives in this spec and in the pull request body; the index link arrives with #1018. | + +## 8. Stop conditions + +* Report `NEEDS_DECISION` rather than widening `kLtx2GoldenTol` if a narrowed + site pushes a golden past it. +* Do not claim any wall-clock or throughput result. There is no host. +* Do not build the NDHWC change here (§5). From aed7d325584b9c01e71038a0196fb42405f41106 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 16 Aug 2026 15:00:34 +0000 Subject: [PATCH 2/3] fix(LTX25-DECODE-DTYPE): the decode computed in f64, and torch computes in f32 even at bf16 (#1008) The LTX-2.5 conv video VAE decode declared `double acc` at 8 sites and cast 29 operands to double. No reference accumulates in f64 anywhere on this path, and until now nothing here could report that: the goldens are vacuous on accumulator width by construction, because gen-ltx2-vae-goldens.py:223 casts every upstream parameter to np.float32 and the oracle therefore ran f32 end to end. This port has been accumulating WIDER than the oracle its own goldens came from. Reading upstream gives the tensor dtype, not the accumulator width, so the width was measured. On a reduction engineered so the two widths separate -- 27 taps over a uniform input with weights [+1e8, 0.1 x 25, -1e8], where half an ulp of 1e8 is 4.0 and the small taps sum to 2.5, so every partial sum rounds back to 1e8 in any f32 order -- F.conv3d returns 0.0 for f32 tensors AND for bf16 tensors, upstream's own dtype, while an f64 accumulator returns 2.5. Every accumulator and every elementwise step on the data path therefore narrows to f32. The epsilons, the once-per-block sqrt(C) scalars and the TimestepEmbedding frequency table stay f64 and each says why at its own site. WIDTH WAS NOT THE WHOLE MIRROR. Narrowing it while keeping the naive serial summation order pushed test_ltx2_tiling's non-causal untiled control to 5.00679e-06 against a 5e-06 tolerance, a real RED. The fix is the ORDER, and it is a closer mirror rather than a looser one: CausalConv3d now keeps one partial sum per input channel, which is how torch's blocked-GEMM f32 convolution sums, and which is why torch.sum returns 2.0999999 on the same reduction where a naive serial f32 sum returns 0.0. No tolerance was touched. With both fixed the numerics essentially do not move: against the f64 arm the shipped arm is 1.07x to 1.31x on every video golden, where the naive f32 arm was 1.9x to 2.9x. The Conv video decoder golden reads 1.40071e-06 (f64), 4.12762e-06 (naive f32), 1.72853e-06 (shipped). Measured by zeroing the tolerance in both suites to read the values, then restoring both. The new case enters through Ltx2VideoDecodeStreaming, the production entry the render path calls, and asserts an analytic consequence rather than a recorded vector. Its conv_out bias of 7 is load-bearing: the first draft expected zero, and the reachability mutation -- deleting the Ltx2ConvVideoDecode call at ltx2_video_vae_tiled.cpp:113 -- PASSED, because a decode that never ran returns zeros too. That first mutation attempt also failed to build while a stale binary printed a plausible verdict, which is why the spec records whether each mutation built. Gating is uneven and the spec says so rather than implying coverage: widening Linear3d back to double is caught by NOTHING, 40/40 cases pass, so nine sites are narrowed on upstream grounding and review rather than on a test, and that gap is owed. No speed number is claimed. dgx.casa was unreachable throughout, so the magnitude is owed and unmeasured. The NDHWC half of the lever is filed rather than half-built: the blocker is not `Volume`, whose At() has 16 call sites, but MiniMaxH3GroupNorm3d, whose signature hard-codes a channel-major buffer and which MiniMax-H3's VAE CNN and the LTX-2 audio VAE also call. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/ltx25-decode-dtype.md | 91 ++++++- docs/FEATURES.md | 1 + .../model_executor/models/ltx2_video_vae.cpp | 226 ++++++++++++------ tests/vllm/models/test_ltx2_vae.cpp | 125 ++++++++++ 4 files changed, 364 insertions(+), 79 deletions(-) diff --git a/.agents/specs/ltx25-decode-dtype.md b/.agents/specs/ltx25-decode-dtype.md index c09865fab..70939045f 100644 --- a/.agents/specs/ltx25-decode-dtype.md +++ b/.agents/specs/ltx25-decode-dtype.md @@ -252,13 +252,100 @@ see and a memory-format change across a shared seam are two independent reviews. | owed | what would settle it | |---|---| | **The speed magnitude of this change. UNMEASURED.** No number is claimed. | One `Ltx2ConvVideoDecode` wall at a fixed size on an idle host, same binary, f64 arm against f32 arm. `dgx.casa` was unreachable for this row's whole duration; any host that can run the decode settles it, and #1010's phase timings make it readable from a render. | +| **A width gate for the nine sites that have none** (§8.2). `Linear3d` was widened back to `double` and **40/40 cases passed**. | One separable-reduction case per site, in the shape §3 established for the convolution. | +| **A blocked summation order for the sites that still sum naively** (§8.1) — `Linear3d`, the attention block, `PixelNorm`. `CausalConv3d` has one; the others do not, and their reductions are shorter but not short. | The same treatment, measured the same way. None of them is near tolerance today. | | NDHWC / `channels_last_3d` (§5) | its own row, after or with a SIMD or device arm | | bf16 *storage*, the phase L6 production arm (`ltx2_video_vae.cpp:46-49`) | [#1007](https://github.com/mudler/vllm.cpp/issues/1007) and the L6 row | | The `.agents/issue-index.md` row for [#1008](https://github.com/mudler/vllm.cpp/issues/1008) | It is **deliberately not appended here.** The row exists on PR #1018, which filed the issue and is unmerged. `.gitattributes:7` sets `merge=union` on that file and `scripts/check-agent-record.py:1437-1442` refuses a duplicate issue number with "duplicate is what two branches appending the same issue look like". Appending it here would turn `main` red for every branch the moment #1018 merges — the exact failure a duplicate #995 row caused. The link lives in this spec and in the pull request body; the index link arrives with #1018. | -## 8. Stop conditions +## 8. Outcome — what was measured + +### 8.1 The numerics moved, and by how much + +They did not stay put, and §4 named this as the risk that could change the +design. Both arms were run with `kLtx2GoldenTol` temporarily set to `0.0` so +every case reports its `max|diff|` rather than only its verdict. Same binary +recipe, same host, one build each. + +| golden arm | f64 acc (before) | f32 acc, NAIVE serial | f32 acc, BLOCKED (shipped) | tol | +|---|---|---|---|---| +| Conv video decoder | 1.40071e-06 | 4.12762e-06 | **1.72853e-06** | 5e-06 | +| non-causal Conv video decoder | 1.81794e-06 | 3.51667e-06 | 2.08616e-06 | 5e-06 | +| norm_eps-binding video decoder | 9.05246e-07 | 1.16974e-06 | 1.54972e-06 | 5e-06 | +| tiled decode, untiled control A | 2.08616e-06 | **5.00679e-06 FAIL** | 2.74181e-06 | 5e-06 | +| tiled decode, untiled control B | 2.62260e-06 | (same case) | 2.80142e-06 | 5e-06 | +| cropped video encoder | 4.17233e-07 | 8.94070e-07 | 4.76837e-07 | 5e-06 | +| video encoder (`*_res`) | 4.17233e-07 | 8.94070e-07 | 4.76837e-07 | 5e-06 | +| causal-arm video encoder | 2.98023e-07 | 3.83705e-07 | 4.17233e-07 | 5e-06 | +| video encoder (strided convs) | 5.96046e-07 | 5.96046e-07 | 8.34465e-07 | 5e-06 | +| all 13 audio arms | unchanged | unchanged | unchanged | untouched by this row | + +**The risk §4 named as design-changing actually bound, and it changed the +design.** Narrowing the width while keeping the naive serial summation order +pushed `test_ltx2_tiling`'s non-causal untiled control to **5.00679e-06 against a +5e-06 tolerance** — over by 0.14%, a genuine RED in the full gate, not a near +miss. The tolerance was not touched. + +**The fix is the summation ORDER, and it is a closer mirror rather than a looser +one.** `CausalConv3d` now keeps one partial sum per input channel and adds the +partials, so a `ci * kernel^3` reduction accumulates error with `sqrt(kernel^3)` +per block instead of `sqrt(ci * kernel^3)` across the whole length. That is what +torch's f32 convolution does — it is a blocked GEMM, which is exactly why §1's +probe found `torch.sum` returning 2.0999999 where a naive serial f32 sum returns +0.0. The width and the order are two separate mirroring questions and this row +had to answer both. + +**The result is that the numerics essentially did not move.** Against the f64 arm +the shipped blocked-f32 arm is 1.07x to 1.31x on every video arm, where the naive +arm was 1.9x to 2.9x. The worst arm sits at 56% of tolerance — a 1.8x margin, +against 1.9x for the f64 arm it replaces and 1.0x for the naive attempt. No +headroom was meaningfully spent, and no tolerance was widened. + +### 8.2 What is gated, and what is not + +Stated plainly because the answer is uneven and a summary would hide it. Each +narrowed site was widened back to `double` on its own, rebuilt, and rerun. + +| mutation | built | exit | detected by | +|---|---|---|---| +| W1 — `CausalConv3d` accumulator widened | yes, 0 errors | 1 | the new width case (1.03473 vs 7) | +| W2 — `Linear3d` accumulator widened | yes, 0 errors | **0** | **nothing. 40/40 cases pass** | +| W3 — `PixelNorm` `mean_sq` widened | yes, 0 errors | 1 | the Conv video decoder golden, incidentally | +| R — the production call site deleted | yes, 0 errors | 1 | the new width case (7 vs 7-minus-nothing) | + +**W2 is an honest gap and it is owed.** Only the convolution's width is gated on +purpose; `PixelNorm` is caught by accident, because a mixed-width path happens to +diverge from torch further than a uniformly f32 one; and `Linear3d`, the +attention block's four accumulators, the timestep Linears and the encoder's group +mean have no gate on their width at all. They are narrowed on upstream grounding +and on review, not on a test. Closing that needs one separable-reduction case per +site, in the shape §3 established. + +### 8.3 The reachability case that passed while measuring nothing + +Worth recording because it nearly shipped. The first draft of the width case +expected **zero**, which is what an f32 accumulator produces on the engineered +reduction. Mutation R replaced the production `Ltx2ConvVideoDecode` call with a +zero-filled buffer — and the case **passed**, because a decode that never ran +produces zeros too. The case measured nothing and reported success. + +The fix is `conv_out.conv.bias = 7`, which moves the expectation off zero: the +f32 arm must return exactly 7, the f64 arm returns 8.03473, and a stub returns 0. +All three are now distinguishable. A recorded value is not a reached one, and an +expectation that coincides with the zero value of an absent computation is not an +assertion. + +The first attempt at mutation R also **failed to build** (3 `-Werror` unused- +parameter errors) while the stale binary still ran and printed a plausible +verdict. Both facts are why every mutation above reports whether it built. + +## 9. Stop conditions * Report `NEEDS_DECISION` rather than widening `kLtx2GoldenTol` if a narrowed - site pushes a golden past it. + site pushes a golden past it. **This fired** (§8.1). It was resolved by + mirroring upstream's summation order, which this spec had already named as the + remedy, rather than by touching the tolerance or restoring the f64 width — so + it is recorded here as a stop condition that triggered and was answered within + the row's own design, not as one that was waived. * Do not claim any wall-clock or throughput result. There is no host. * Do not build the NDHWC change here (§5). diff --git a/docs/FEATURES.md b/docs/FEATURES.md index d5784a9dc..1163a0764 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -168,6 +168,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`, diffusers lane) | MiniMax-Music3 (8.6B Qwen3 LLM + 0.646B RVQ decoder + 2.4B fp32 DiT + DAC Flow-VAE); diffusers arm, ~28.5 GB | `ACTIVE`. Loader 1413/1413; AR, acoustic and the 8.6B LM forward all gated vs real weights; `SpeechRegistry` + `vllm_speech_*` v20 + `/v1/audio/speech`; GGUF Q4_K depth decoder value-gated. HTTP request OBSERVED (#852) | Not measured. The denominator will be SGLang-Omni in its production configuration (both CUDA graphs, compiled DIT and DAV, batched seeded sampling) | | LTX-2.5 DFR base + generated keyframe slots | LTX-2.5 (21.00B video+audio) | gated vs EXECUTED upstream `dfr_layout` + 3 `dfr_pipeline` helpers @ `fd4ded7f` (`test_ltx2_dfr` 11/11, 652 assertions); canvas, tiles, stitch, carry-forward as EXACT index vectors, since each defect is plausible| `--pipeline-kind dfr`. Canvas PADS 9 to 25 then trims back; slots on the x8 grid, MARKED, read back BEFORE the trim. `num_generated_keyframes` SERVED elsewhere. Temporal ROUNDS refused (#986); detail LoRA refused (#975)| | LTX-2.5 tiled + streaming Conv VAE decode | LTX-2.5 video VAE | gated vs executed upstream `ltx_core` @ `fd4ded7f` (`test_ltx2_tiling` 10/10, 915 assertions); one-tile and untiled-spatial controls BIT-EXACT vs untiled on both causality arms; an untiled frames axis is REFUSED | Streams temporal chunks through upstream's AUTO layout (768/64 px, 80/24 frames); above one tile the pixel volume is never materialized. NO-OP below 768px and 81 frames; 81-120 IS tiled, differing 6.70% of range | +| LTX-2.5 Conv VAE decode arithmetic width | LTX-2.5 video VAE | `test_ltx2_vae` "the decode's convolution accumulates in f32", entering through `Ltx2VideoDecodeStreaming`; widening the accumulator to `double`, or deleting the production call site, each turns it RED | **f32**, the width `F.conv3d` uses at f32 AND bf16 (MEASURED). Was f64 at 8 sites ([#1008](https://github.com/mudler/vllm.cpp/issues/1008)). Conv sums BLOCKED per input channel, as torch's. STORAGE stays f32; bf16 owed | | LTX-2.5 retake (`RetakePipeline`, regenerate a time window) | LTX-2.5 DiT + video VAE encoder | `test_ltx2_retake` 4/4 (69 assertions) and 4 `test_ltx2_video` cases entering through `Generate`; mask, conform and the four-way plan pinned to upstream `fd4ded7f` | `--pipeline-kind retake` on `ltx2-gen`. Source is a `frame_%06d.ppm` DIRECTORY; a container is REFUSED (no demuxer). Geometry comes from the clip. A folder has no audio, so the soundtrack is generated | | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) | | DFlash block-diffusion | Qwen3 (DFlash draft) | near-tie e2e 27/27 vs vLLM | 2.9x over spec-off, 1.003x vs vLLM DFlash-on | diff --git a/src/vllm/model_executor/models/ltx2_video_vae.cpp b/src/vllm/model_executor/models/ltx2_video_vae.cpp index ad7880b72..dafc39b9c 100644 --- a/src/vllm/model_executor/models/ltx2_video_vae.cpp +++ b/src/vllm/model_executor/models/ltx2_video_vae.cpp @@ -43,6 +43,23 @@ // every upstream parameter to f32, so the oracle itself runs f32 and a dtype // comparison against it is vacuous by construction. // +// ─── THE ARITHMETIC IS f32 TOO, AND USED NOT TO BE (#1008) ─────────────────── +// Storage being f32 says nothing about the width the arithmetic runs at, and +// until #1008 this file accumulated every convolution, GEMM, norm and softmax in +// `double` — a width no reference uses anywhere on this path. Upstream's ops are +// plain `nn.Conv3d` / `nn.Conv2d` / `F.normalize` / SDPA, which accumulate in the +// tensor dtype. That was MEASURED rather than assumed: on a reduction engineered +// so the widths separate, `F.conv3d` returns 0.0 for f32 AND for bf16 tensors +// while an f64 accumulator returns 2.5. The case +// "the decode's convolution accumulates in f32" in tests/vllm/models/ +// test_ltx2_vae.cpp is that instrument, and it is the only gate here that can +// see the width — for the reason the paragraph above gives. +// +// What deliberately stays f64, each annotated at its site: the pinned config +// epsilons, the once-per-block scalars `sqrt(C)` and `1/sqrt(C)`, and the +// TimestepEmbedding frequency table, which is a constant precompute rather than +// a data path. +// // PHASE L6 OWES THE PRODUCTION ARM — the bf16/NVFP4 decode that inherits the // checkpoint dtype the way upstream does. Until it lands, this file is a // correctness reference, not the shipping path, and no memory or throughput @@ -162,23 +179,38 @@ Volume CausalConv3d(const Volume& in, int64_t out_channels, int64_t kernel, bool for (int64_t ti = 0; ti < out.t; ++ti) { for (int64_t hi = 0; hi < out.h; ++hi) { for (int64_t wi = 0; wi < out.w; ++wi) { - double acc = bias != nullptr ? (*bias)[static_cast(oc)] : 0.0; + // f32, because that is the width `nn.Conv3d` accumulates in — MEASURED, + // not assumed: F.conv3d returns 0.0 on the separable reduction in + // tests/vllm/models/test_ltx2_vae.cpp for f32 AND for bf16 tensors, + // where an f64 accumulator returns 2.5 (#1008). + float acc = bias != nullptr ? (*bias)[static_cast(oc)] : 0.0f; for (int64_t ic = 0; ic < ci; ++ic) { + // BLOCKED, one partial sum per input channel, and this is the ORDER + // as well as the width. A single naive serial f32 sum over all + // `ci * kernel^3` taps accumulates error with sqrt of the whole + // length; splitting it into `ci` blocks of `kernel^3` accumulates + // with sqrt of each. That is not a local optimisation — torch's f32 + // convolution is a blocked GEMM and sums exactly this way, which is + // why `torch.sum` on the separable reduction returns 2.0999999 where + // a naive serial f32 sum returns 0.0. Narrowing the width alone, + // with the naive order kept, pushed the non-causal tiled golden to + // 5.00679e-06 against a 5e-06 tolerance — MEASURED, and the reason + // this loop is shaped this way. + float tap = 0.0f; for (int64_t a = 0; a < kernel; ++a) { for (int64_t b = 0; b < kernel; ++b) { for (int64_t d = 0; d < kernel; ++d) { - acc += static_cast( - padded[static_cast(((ic * pt + ti * stride_t + a) * ph + - hi * stride_h + b) * - pw + - wi * stride_w + d)]) * - static_cast(weight[static_cast( - (((oc * ci + ic) * kernel + a) * kernel + b) * kernel + d)]); + tap += padded[static_cast( + ((ic * pt + ti * stride_t + a) * ph + hi * stride_h + b) * pw + + wi * stride_w + d)] * + weight[static_cast( + (((oc * ci + ic) * kernel + a) * kernel + b) * kernel + d)]; } } } + acc += tap; } - out.data[out.At(oc, ti, hi, wi)] = static_cast(acc); + out.data[out.At(oc, ti, hi, wi)] = acc; } } } @@ -198,19 +230,25 @@ Volume Linear3d(const Volume& in, int64_t out_channels, const std::vector const int64_t n = in.spatial(); for (int64_t oc = 0; oc < out_channels; ++oc) { for (int64_t i = 0; i < n; ++i) { - double acc = bias[static_cast(oc)]; + // f32: this is an `nn.Conv3d` upstream too (make_linear_nd's dims==3 + // branch, convolution.py:84-85), so it accumulates at the same width as + // every other conv on the path. + float acc = bias[static_cast(oc)]; for (int64_t ic = 0; ic < in.channels; ++ic) { - acc += static_cast(in.data[static_cast(ic * n + i)]) * - static_cast(weight[static_cast(oc * in.channels + ic)]); + acc += in.data[static_cast(ic * n + i)] * + weight[static_cast(oc * in.channels + ic)]; } - out.data[static_cast(oc * n + i)] = static_cast(acc); + out.data[static_cast(oc * n + i)] = acc; } } return out; } void Silu(std::vector& x) { - for (float& v : x) v = static_cast(v / (1.0 + std::exp(-static_cast(v)))); + // f32: `F.silu` computes in the activation dtype. `std::exp(-v)` on a float + // selects the float overload — spelling it `-static_cast(v)` would + // quietly restore the f64 path this deliberately leaves. + for (float& v : x) v = v / (1.0f + std::exp(-v)); } // PixelNorm() with its DEFAULT eps of 1e-8 (normalization.py:22, reached bare @@ -218,16 +256,22 @@ void Silu(std::vector& x) { // audio VAE gets through build_normalization_layer. void PixelNorm(std::vector& x, int64_t channels, int64_t spatial, double eps) { for (int64_t i = 0; i < spatial; ++i) { - double mean_sq = 0.0; + // f32: `torch.mean(x**2, dim=...)` runs in the activation dtype + // (normalization.py:37-40). + float mean_sq = 0.0f; for (int64_t c = 0; c < channels; ++c) { - const double v = x[static_cast(c * spatial + i)]; + const float v = x[static_cast(c * spatial + i)]; mean_sq += v * v; } - mean_sq /= static_cast(channels); - const double inv = 1.0 / std::sqrt(mean_sq + eps); + mean_sq /= static_cast(channels); + // The reciprocal is f32 too: upstream is `x / torch.sqrt(mean_sq + eps)` + // with every term in the tensor dtype (normalization.py:37-40). `eps` + // remains an f64 PARAMETER because it is a pinned config threshold + // (Ltx2ConvVideoDecoderConfig::pixel_norm_eps); it is narrowed here, at the + // one point it enters the arithmetic. + const float inv = 1.0f / std::sqrt(mean_sq + static_cast(eps)); for (int64_t c = 0; c < channels; ++c) { - x[static_cast(c * spatial + i)] = - static_cast(x[static_cast(c * spatial + i)] * inv); + x[static_cast(c * spatial + i)] = x[static_cast(c * spatial + i)] * inv; } } } @@ -281,6 +325,12 @@ std::vector TimestepEmbedding(double timestep, int64_t embedding_dim, constexpr int64_t kProjChannels = 256; constexpr double kMaxPeriod = 10000.0; const int64_t half = kProjChannels / 2; + // DELIBERATE f64 EXCEPTION, and the only one on this path. `proj` is a + // transcendental CONSTANT table — 256 cos/sin values built once per block from + // the timestep, never a per-element data-path accumulation — so it is off + // every hot path, and evaluating it in f64 sits closer to the exact value that + // upstream's f32 `torch.arange` table approximates. Everything downstream of + // it is f32 (#1008). std::vector proj(static_cast(kProjChannels)); for (int64_t i = 0; i < half; ++i) { // downscale_freq_shift = 0, so the divisor is exactly half_dim. @@ -298,23 +348,25 @@ std::vector TimestepEmbedding(double timestep, int64_t embedding_dim, VT_CHECK(static_cast(w1.size()) == embedding_dim * kProjChannels, "ltx2 timestep embedding: linear_1 shape does not match the embedding dim"); - std::vector hidden(static_cast(embedding_dim)); + // f32 for both `nn.Linear` accumulators and for the hidden activation between + // them: upstream's TimestepEmbedder is two plain Linears with a SiLU, all in + // the activation dtype. The frequency table above stays f64 — see its note. + std::vector hidden(static_cast(embedding_dim)); for (int64_t o = 0; o < embedding_dim; ++o) { - double acc = b1[static_cast(o)]; + float acc = b1[static_cast(o)]; for (int64_t i = 0; i < kProjChannels; ++i) { - acc += proj[static_cast(i)] * - static_cast(w1[static_cast(o * kProjChannels + i)]); + acc += static_cast(proj[static_cast(i)]) * + w1[static_cast(o * kProjChannels + i)]; } - hidden[static_cast(o)] = acc / (1.0 + std::exp(-acc)); // SiLU + hidden[static_cast(o)] = acc / (1.0f + std::exp(-acc)); // SiLU } std::vector out(static_cast(embedding_dim)); for (int64_t o = 0; o < embedding_dim; ++o) { - double acc = b2[static_cast(o)]; + float acc = b2[static_cast(o)]; for (int64_t i = 0; i < embedding_dim; ++i) { - acc += hidden[static_cast(i)] * - static_cast(w2[static_cast(o * embedding_dim + i)]); + acc += hidden[static_cast(i)] * w2[static_cast(o * embedding_dim + i)]; } - out[static_cast(o)] = static_cast(acc); + out[static_cast(o)] = acc; } return out; } @@ -330,12 +382,12 @@ void FeedSpatialNoise(Volume& x, const std::vector& per_channel_scale, VT_CHECK(static_cast(plane.size()) == x.h * x.w, "ltx2 video vae: the noise stream returned the wrong element count"); for (int64_t c = 0; c < x.channels; ++c) { - const double scale = per_channel_scale[static_cast(c)]; + // f32: upstream scales and adds the noise plane in the activation dtype. + const float scale = per_channel_scale[static_cast(c)]; for (int64_t ti = 0; ti < x.t; ++ti) { for (int64_t hi = 0; hi < x.h; ++hi) { for (int64_t wi = 0; wi < x.w; ++wi) { - x.data[x.At(c, ti, hi, wi)] += static_cast( - static_cast(plane[static_cast(hi * x.w + wi)]) * scale); + x.data[x.At(c, ti, hi, wi)] += plane[static_cast(hi * x.w + wi)] * scale; } } } @@ -353,13 +405,15 @@ void ApplyAdaLn(Volume& x, const std::vector& table, const std::vector(table[static_cast(shift_row * c + ch)]) + - static_cast(embed[static_cast(shift_row * c + ch)]); - const double scale = static_cast(table[static_cast(scale_row * c + ch)]) + - static_cast(embed[static_cast(scale_row * c + ch)]); + // f32: upstream adds two f32 tensors and applies `x * (1 + scale) + shift` + // in the activation dtype (resnet.py:135-147). + const float shift = table[static_cast(shift_row * c + ch)] + + embed[static_cast(shift_row * c + ch)]; + const float scale = table[static_cast(scale_row * c + ch)] + + embed[static_cast(scale_row * c + ch)]; for (int64_t i = 0; i < n; ++i) { - x.data[static_cast(ch * n + i)] = static_cast( - static_cast(x.data[static_cast(ch * n + i)]) * (1.0 + scale) + shift); + x.data[static_cast(ch * n + i)] = + x.data[static_cast(ch * n + i)] * (1.0f + scale) + shift; } } } @@ -515,73 +569,87 @@ Volume AttnBlock3d(const Ltx2VaeWeights& weights, const std::string& prefix, con const double attn_scale = 1.0 / std::sqrt(static_cast(c)); Volume out = x; - std::vector normed(static_cast(c * n)); - std::vector q(static_cast(c * n)), k(static_cast(c * n)), + // f32 activations, not f64. Upstream holds q/k/v and the attention output in + // the tensor dtype (attention.py:63-67) and never promotes; these six buffers + // are the block's whole scratch footprint, so the width is bytes as well as + // arithmetic. `norm_scale` and `attn_scale` above stay f64 — upstream's + // `channels**0.5` is a Python float evaluated once per block. + std::vector normed(static_cast(c * n)); + std::vector q(static_cast(c * n)), k(static_cast(c * n)), v(static_cast(c * n)); - std::vector scores(static_cast(n)); - std::vector attended(static_cast(c * n)); + std::vector scores(static_cast(n)); + std::vector attended(static_cast(c * n)); for (int64_t frame = 0; frame < x.t; ++frame) { // _RMSNorm2D: F.normalize(x, dim=1) * (sqrt(C) * gamma) — an L2 normalize with // torch's 1e-12 floor, not a mean-square RMS. for (int64_t i = 0; i < n; ++i) { - double sum_sq = 0.0; + // f32: `F.normalize(x, dim=1)` computes its norm in the input dtype + // (attention.py:23). torch's 1e-12 floor stays f64 — it is a threshold. + float sum_sq = 0.0f; for (int64_t ch = 0; ch < c; ++ch) { - const double value = x.data[x.At(ch, frame, i / x.w, i % x.w)]; + const float value = x.data[x.At(ch, frame, i / x.w, i % x.w)]; sum_sq += value * value; } - const double inv = 1.0 / std::max(std::sqrt(sum_sq), kLtx2RmsNorm2dEps); + const float inv = static_cast( + 1.0 / std::max(std::sqrt(static_cast(sum_sq)), kLtx2RmsNorm2dEps)); + // Same left-to-right association the f64 arm used; only the width changes. + const float norm_scale_f = static_cast(norm_scale); for (int64_t ch = 0; ch < c; ++ch) { - normed[static_cast(ch * n + i)] = - x.data[x.At(ch, frame, i / x.w, i % x.w)] * inv * norm_scale * - static_cast(gamma[static_cast(ch)]); + normed[static_cast(ch * n + i)] = x.data[x.At(ch, frame, i / x.w, i % x.w)] * inv * + norm_scale_f * gamma[static_cast(ch)]; } } // to_qkv is a 1x1 Conv2d emitting [q | k | v] along the channel axis, and the // rearrange to tokens keeps that split on the LAST axis (attention.py:63-64). + // f32: `to_qkv` is a 1x1 nn.Conv2d (attention.py:55), the same accumulator + // width as every other conv here. for (int64_t oc = 0; oc < 3 * c; ++oc) { - std::vector& dst = oc < c ? q : (oc < 2 * c ? k : v); + std::vector& dst = oc < c ? q : (oc < 2 * c ? k : v); const int64_t row = oc % c; for (int64_t i = 0; i < n; ++i) { - double acc = qkv_b[static_cast(oc)]; + float acc = qkv_b[static_cast(oc)]; for (int64_t ic = 0; ic < c; ++ic) { - acc += normed[static_cast(ic * n + i)] * - static_cast(qkv_w[static_cast(oc * c + ic)]); + acc += normed[static_cast(ic * n + i)] * qkv_w[static_cast(oc * c + ic)]; } dst[static_cast(row * n + i)] = acc; } } + // f32: SDPA computes scores, softmax and the value-weighted sum in the + // tensor dtype (attention.py:65). `attn_scale` stays f64 for the same reason + // `norm_scale` does. + const float attn_scale_f = static_cast(attn_scale); for (int64_t i = 0; i < n; ++i) { - double max_score = -std::numeric_limits::infinity(); + float max_score = -std::numeric_limits::infinity(); for (int64_t j = 0; j < n; ++j) { - double dot = 0.0; + float dot = 0.0f; for (int64_t ch = 0; ch < c; ++ch) { dot += q[static_cast(ch * n + i)] * k[static_cast(ch * n + j)]; } - scores[static_cast(j)] = dot * attn_scale; + scores[static_cast(j)] = dot * attn_scale_f; max_score = std::max(max_score, scores[static_cast(j)]); } - double sum = 0.0; + float sum = 0.0f; for (int64_t j = 0; j < n; ++j) { scores[static_cast(j)] = std::exp(scores[static_cast(j)] - max_score); sum += scores[static_cast(j)]; } for (int64_t ch = 0; ch < c; ++ch) { - double acc = 0.0; + float acc = 0.0f; for (int64_t j = 0; j < n; ++j) { acc += scores[static_cast(j)] * v[static_cast(ch * n + j)]; } attended[static_cast(ch * n + i)] = acc / sum; } } + // f32: `proj` is a 1x1 nn.Conv2d (attention.py:56). for (int64_t oc = 0; oc < c; ++oc) { for (int64_t i = 0; i < n; ++i) { - double acc = proj_b[static_cast(oc)]; + float acc = proj_b[static_cast(oc)]; for (int64_t ic = 0; ic < c; ++ic) { - acc += attended[static_cast(ic * n + i)] * - static_cast(proj_w[static_cast(oc * c + ic)]); + acc += attended[static_cast(ic * n + i)] * proj_w[static_cast(oc * c + ic)]; } - out.data[out.At(oc, frame, i / x.w, i % x.w)] += static_cast(acc); + out.data[out.At(oc, frame, i / x.w, i % x.w)] += acc; } } } @@ -625,10 +693,12 @@ Ltx2VideoFrames Ltx2ConvVideoDecode(const Ltx2ConvVideoDecoderConfig& config, const std::vector drawn = noise->Draw(static_cast(x.data.size())); VT_CHECK(drawn.size() == x.data.size(), "ltx2 video vae: the noise stream returned the wrong element count"); + // f32: the blend runs in the activation dtype upstream. The two scalars are + // config values, so they are narrowed once rather than per element. + const float noise_scale = static_cast(config.decode_noise_scale); + const float keep_scale = static_cast(1.0 - config.decode_noise_scale); for (size_t i = 0; i < x.data.size(); ++i) { - x.data[i] = static_cast( - static_cast(drawn[i]) * config.decode_noise_scale + - (1.0 - config.decode_noise_scale) * static_cast(x.data[i])); + x.data[i] = drawn[i] * noise_scale + keep_scale * x.data[i]; } } { @@ -640,12 +710,13 @@ Ltx2VideoFrames Ltx2ConvVideoDecode(const Ltx2ConvVideoDecoderConfig& config, static_cast(mean_of_means.size()) == latent_channels, "ltx2 video vae: per-channel statistics must have one value per latent channel"); const int64_t n = x.spatial(); + // f32: upstream's de-normalize is `latent * std + mean` on f32/bf16 tensors. for (int64_t c = 0; c < latent_channels; ++c) { + const float std_c = std_of_means[static_cast(c)]; + const float mean_c = mean_of_means[static_cast(c)]; for (int64_t i = 0; i < n; ++i) { - x.data[static_cast(c * n + i)] = static_cast( - static_cast(x.data[static_cast(c * n + i)]) * - static_cast(std_of_means[static_cast(c)]) + - static_cast(mean_of_means[static_cast(c)])); + x.data[static_cast(c * n + i)] = + x.data[static_cast(c * n + i)] * std_c + mean_c; } } } @@ -913,13 +984,12 @@ Volume SpaceToDepthDownsample(const VideoConvSpec& spec, const Ltx2VaeWeights& w const int64_t n = skip.spatial(); for (int64_t c = 0; c < out_channels; ++c) { for (int64_t i = 0; i < n; ++i) { - double acc = 0.0; + // f32: upstream's group mean runs in the activation dtype. + float acc = 0.0f; for (int64_t g = 0; g < group_size; ++g) { - acc += static_cast( - folded_in.data[static_cast((c * group_size + g) * n + i)]); + acc += folded_in.data[static_cast((c * group_size + g) * n + i)]; } - skip.data[static_cast(c * n + i)] = - static_cast(acc / static_cast(group_size)); + skip.data[static_cast(c * n + i)] = acc / static_cast(group_size); } } @@ -1122,12 +1192,14 @@ Ltx2LatentVolume Ltx2ConvVideoEncode(const Ltx2ConvVideoEncoderConfig& config, out.width = x.w; out.data.resize(static_cast(out.elems())); const int64_t elems = x.spatial(); + // f32: the encoder's normalize is the decoder de-normalize run backwards, and + // upstream computes it in the activation dtype on both sides. for (int64_t c = 0; c < latent_channels; ++c) { - const double mean = mean_of_means[static_cast(c)]; - const double denom = std_of_means[static_cast(c)]; + const float mean = mean_of_means[static_cast(c)]; + const float denom = std_of_means[static_cast(c)]; for (int64_t i = 0; i < elems; ++i) { - out.data[static_cast(c * elems + i)] = static_cast( - (static_cast(x.data[static_cast(c * elems + i)]) - mean) / denom); + out.data[static_cast(c * elems + i)] = + (x.data[static_cast(c * elems + i)] - mean) / denom; } } return out; diff --git a/tests/vllm/models/test_ltx2_vae.cpp b/tests/vllm/models/test_ltx2_vae.cpp index 12c8c2e0a..b3f688d41 100644 --- a/tests/vllm/models/test_ltx2_vae.cpp +++ b/tests/vllm/models/test_ltx2_vae.cpp @@ -26,6 +26,9 @@ #include "vllm/model_executor/models/ltx2_audio_vae.h" #include "vllm/model_executor/models/ltx2_audio_vae_encoder.h" #include "vllm/model_executor/models/ltx2_conditioning.h" +// The accumulator-width case enters the decode through the PRODUCTION streaming +// entry point rather than through Ltx2ConvVideoDecode (issue #1008). +#include "vllm/model_executor/models/ltx2_tiling.h" #include "vllm/model_executor/models/ltx2_video_vae.h" #include "vllm/model_executor/models/ltx2_video_vae_encoder.h" // vocoder1d::kSnakeEps: the Snake/SnakeBeta stabilizer is SHARED with MiniMax-H3's @@ -1059,6 +1062,128 @@ TEST_CASE("ltx2 vae: the NON-causal Conv video decoder matches upstream ltx_core CHECK(causal_frames.data != frames.data); } +TEST_CASE("ltx2 vae: the decode's convolution accumulates in f32, the width torch uses") { + // THE ONE DEFECT THE GOLDENS STRUCTURALLY CANNOT REPORT, so it gets its own + // instrument (issue #1008, .agents/specs/ltx25-decode-dtype.md). + // + // Every golden in this file is vacuous on accumulator WIDTH by construction: + // scripts/gen-ltx2-vae-goldens.py:223 casts every upstream parameter with + // `values.astype(np.float32)`, so the oracle ran f32 end to end and a dtype + // comparison against it compares nothing. A decode that accumulated in f64 — + // as this one did until #1008 — stays numerically plausible, keeps every + // golden green, and moves twice the bytes. AGENTS.md says a token gate cannot + // detect a dtype that is too wide; this case is what detects it here. + // + // THE REDUCTION IS ENGINEERED SO THE TWO WIDTHS ARE SEPARABLE, and the + // expected value is UPSTREAM'S, not a recording of what this port emits. + // MEASURED with torch 2.11.0 on exactly the taps below: + // F.conv3d, f32 tensors -> 0.0 + // F.conv3d, bf16 tensors -> 0.0 (upstream's own dtype, distilled.py:109) + // F.conv3d, f64 tensors -> 2.500000014901161 + // Why 0.0 holds for ANY f32 summation order: half an ulp of 1e8 is 4.0, and + // the 25 small taps sum to 2.5, so every partial sum rounds back to exactly + // 1e8 and the closing -1e8 cancels it exactly. An f64 accumulator keeps the + // 2.5. There is no order in which an f32 accumulator does. + constexpr float kBig = 1e8f; + constexpr float kSmall = 0.1f; + + vllm::Ltx2ConvVideoDecoderConfig cfg; + cfg.prefix = "ltx2.videodec.accwidth."; + cfg.in_channels = 1; + cfg.out_channels = 1; + cfg.patch_size = 1; + cfg.base_channels = 2; + cfg.causal = false; + cfg.timestep_conditioning = false; + cfg.norm_layer = vllm::Ltx2NormLayer::kPixelNorm; + // REPLICATE, not the fixtures' reflect: with a uniform input every conv tap + // then reads exactly 1.0 at every output voxel INCLUDING the borders, so one + // reduction is repeated everywhere and no voxel is a special case. `kZeros` + // would zero the border taps and break the derivation below. + cfg.spatial_padding_mode = vllm::Ltx2PaddingMode::kReplicate; + // No blocks at all. The decode is then conv_in -> PixelNorm -> SiLU -> + // conv_out, which is short enough to carry an ANALYTIC expectation end to end + // rather than a checked-in vector that only records today's behaviour. + cfg.decoder_blocks = {}; + + const std::string p = cfg.prefix; + vllm::Ltx2VaeWeights weights; + weights.tensors[p + "per_channel_statistics.std-of-means"] = {1.0f}; + weights.tensors[p + "per_channel_statistics.mean-of-means"] = {0.0f}; + + // conv_in is [out=2, in=1, 3, 3, 3]. Output channel 0 carries the separable + // reduction in the exact order CausalConv3d walks it (ic, then a, b, d, so + // flat 0..26); output channel 1 is all-zero with a bias of 1, purely to give + // PixelNorm a second channel with a known norm. + std::vector conv_in(2 * 1 * 27, 0.0f); + conv_in[0] = kBig; + for (size_t i = 1; i < 26; ++i) conv_in[i] = kSmall; + conv_in[26] = -kBig; + weights.tensors[p + "conv_in.conv.weight"] = conv_in; + weights.tensors[p + "conv_in.conv.bias"] = {0.0f, 1.0f}; + + // conv_out is [out=1, in=2, 3, 3, 3]; it selects channel 0's centre tap and + // nothing else, so it forwards the value under test without adding a second + // reduction that could mask it. + // + // THE BIAS IS 7, AND THAT IS NOT COSMETIC. With a bias of 0 the f32 answer is + // zero, and a decode that never ran — a deleted call site handing back a + // zero-filled buffer — produces zero as well, so the case would pass while + // measuring nothing. This was not hypothetical: the reachability mutation was + // run, the stub returned zeros, and an earlier draft of this case PASSED. + // Offsetting the expectation off zero is what separates "accumulated in f32" + // from "nothing reached this at all". + constexpr float kOutBias = 7.0f; + std::vector conv_out(1 * 2 * 27, 0.0f); + conv_out[13] = 1.0f; // ic=0, a=b=d=1 + weights.tensors[p + "conv_out.conv.weight"] = conv_out; + weights.tensors[p + "conv_out.conv.bias"] = {kOutBias}; + + const int64_t lt = 1, lh = 2, lw = 2; + const std::vector latent(static_cast(lt * lh * lw), 1.0f); + + vllm::Ltx2TileSizeConfig tiling; + tiling.frames = vllm::Ltx2DimensionSizeConfig{10000, 0}; + tiling.height = vllm::Ltx2DimensionSizeConfig{10000, 0}; + tiling.width = vllm::Ltx2DimensionSizeConfig{10000, 0}; + + // ENTERS THROUGH THE PRODUCTION ENTRY POINT. `Ltx2VideoDecodeStreaming` is + // what the render path calls (src/vllm/multimodal/ltx2_video.cpp:3258), and it + // reaches Ltx2ConvVideoDecode through ltx2_video_vae_tiled.cpp:113. Deleting + // that call site turns this case RED, which is the reachability proof; a case + // that called Ltx2ConvVideoDecode directly would prove only that the function + // works. A null noise stream is deliberate: this configuration must not draw, + // and the decode's own VT_CHECK fails loudly if that ever changes. + int64_t chunks = 0; + vllm::Ltx2VideoFrames got; + vllm::Ltx2VideoDecodeStreaming( + vllm::Ltx2VideoDecoderKind::kConv, cfg, weights, latent, cfg.in_channels, lt, lh, lw, + /*noise=*/nullptr, tiling, [&](const vllm::Ltx2VideoChunk& chunk) { + ++chunks; + got = chunk.frames; + }); + REQUIRE(chunks == 1); + REQUIRE(got.data.size() == 4u); + + // THE DERIVATION, so a reader can check the number rather than trust it. + // conv_in ch0 = 0 (f32) or 2.5 (f64); conv_in ch1 = bias = 1 + // PixelNorm over 2 channels: f32 mean_sq = 0.5, and 0 * anything = 0 + // SiLU(0) = 0; conv_out forwards ch0 and adds its bias + // so an f32 accumulator makes EVERY output element exactly the bias. An f64 + // one gives 2.5 -> 2.5/sqrt(3.625) = 1.31306 -> SiLU -> ~1.03473 on top of it, + // a gap of 1.03, which is 200000x the golden tolerance. Nothing between the + // two arms is a matter of tolerance. + const std::vector want(4, kOutBias); + const double err = MaxAbsDiff(got.data, want.data(), got.data.size()); + INFO("conv accumulator width probe max|out - bias| = " << err); + CHECK(err <= kLtx2GoldenTol); + + // The f64 answer stated as a value, so this case fails LOUDLY rather than + // drifting if the accumulator is ever widened again. 1.03473 is what this + // decode returned before #1008 narrowed it. + CHECK(err < 1.0); +} + TEST_CASE("ltx2 vae: the video decoder's norm_eps is gated where it BINDS") { // THE ARM THAT MAKES `Ltx2ConvVideoDecoderConfig::norm_eps` NUMERICALLY // REACHABLE, and the correction of a record that said it was not. From a034b4dd0db6c9bea3aa3a32c27d02a89a5bb791 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 16 Aug 2026 15:07:36 +0000 Subject: [PATCH 3/3] docs(LTX25-DECODE-DTYPE): re-derive the spec's own anchors at the final tree Three `ltx2_video_vae.cpp` anchors in the spec were written against the base and went stale inside this pull request, because the change adds a dtype block to the file header and shifts everything below it. `Volume` moved 68-76 -> 85-93 and the phase L6 note 46-49 -> 63-66, both re-derived by searching for the claim's own text rather than by reading the cited span. The defect-site table in section 2 keeps its base anchors deliberately, since it locates the f64 sites as they stood before this change removed them, and it says so. Section 5's bypass-site list now says which basis it uses instead of leaving a reader to assume. The unchanged anchors were re-checked rather than trusted: ltx2_video_vae_tiled.cpp:113 still holds the sole Ltx2ConvVideoDecode call, and ltx2_video.cpp:3258 still holds the Ltx2VideoDecodeStreaming entry. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/ltx25-decode-dtype.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/.agents/specs/ltx25-decode-dtype.md b/.agents/specs/ltx25-decode-dtype.md index 70939045f..de4b29da5 100644 --- a/.agents/specs/ltx25-decode-dtype.md +++ b/.agents/specs/ltx25-decode-dtype.md @@ -27,7 +27,7 @@ nothing. **Not in scope, and deliberately so.** The *storage* dtype stays f32. Upstream stores bf16 (§1), and moving this file to bf16 storage is the production arm -that `ltx2_video_vae.cpp:46-49` already books as phase L6 debt. That is a +that `ltx2_video_vae.cpp:63-66` already books as phase L6 debt. That is a different change with a different risk profile, and this row does not take it. Narrowing the *arithmetic* is separable from narrowing the *storage*, and only the first can be done without a second dtype for every buffer. @@ -195,7 +195,7 @@ bf16 (§1), so the case gates a mirrored property and not a local convention. as `Volume` would be wrong.** `Volume` is a file-local struct in an anonymous namespace -(`ltx2_video_vae.cpp:68-76`) whose `At()` is a single indexing function with 16 +(`ltx2_video_vae.cpp:85-93`) whose `At()` is a single indexing function with 16 call sites. That much is cheap. What is not cheap: 1. **A shared helper hard-codes NCDHW in its signature.** @@ -209,7 +209,8 @@ call sites. That much is cheap. What is not cheap: which spends what the layout was meant to save. This is a shared-seam decision, not a local one. 2. **Ten layout-dependent sites bypass `At()`** with open-coded `c * n + i` - arithmetic — `Linear3d` (`:203,:206`), `ApplyAdaLn` (`:361-362`), the + arithmetic, at §2's `ff264cb82` base anchors rather than this section's + final-tree ones — `Linear3d` (`:203,:206`), `ApplyAdaLn` (`:361-362`), the denormalize (`:645-648`), `SpaceToDepthDownsample` (`:919,:921`), the encoder normalize (`:1129-1130`) — plus the local scratch buffers `padded`, `normed`, `q`, `k`, `v` and `attended`, which carry their own layouts and @@ -255,7 +256,7 @@ see and a memory-format change across a shared seam are two independent reviews. | **A width gate for the nine sites that have none** (§8.2). `Linear3d` was widened back to `double` and **40/40 cases passed**. | One separable-reduction case per site, in the shape §3 established for the convolution. | | **A blocked summation order for the sites that still sum naively** (§8.1) — `Linear3d`, the attention block, `PixelNorm`. `CausalConv3d` has one; the others do not, and their reductions are shorter but not short. | The same treatment, measured the same way. None of them is near tolerance today. | | NDHWC / `channels_last_3d` (§5) | its own row, after or with a SIMD or device arm | -| bf16 *storage*, the phase L6 production arm (`ltx2_video_vae.cpp:46-49`) | [#1007](https://github.com/mudler/vllm.cpp/issues/1007) and the L6 row | +| bf16 *storage*, the phase L6 production arm (`ltx2_video_vae.cpp:63-66`) | [#1007](https://github.com/mudler/vllm.cpp/issues/1007) and the L6 row | | The `.agents/issue-index.md` row for [#1008](https://github.com/mudler/vllm.cpp/issues/1008) | It is **deliberately not appended here.** The row exists on PR #1018, which filed the issue and is unmerged. `.gitattributes:7` sets `merge=union` on that file and `scripts/check-agent-record.py:1437-1442` refuses a duplicate issue number with "duplicate is what two branches appending the same issue look like". Appending it here would turn `main` red for every branch the moment #1018 merges — the exact failure a duplicate #995 row caused. The link lives in this spec and in the pull request body; the index link arrives with #1018. | ## 8. Outcome — what was measured