fix(LTX25-DECODE-DTYPE): the decode computed in f64, and torch computes in f32 even at bf16 (#1008) by localai-bot · Pull Request #1036 · mudler/vllm.cpp

localai-bot · 2026-08-16T15:08:36Z

Lever 2 of the LTX25-DECODE-SPEED investigation (#1006, PR #1018), dtype half.
Closes #1008.

The LTX-2.5 conv video VAE decode declared double acc at 8 sites and cast 29
operands to double. No reference accumulates in f64 anywhere on this path, and
until now nothing here could report that: the goldens are vacuous on accumulator
width by construction, because gen-ltx2-vae-goldens.py:223 casts every upstream
parameter to np.float32 and the oracle therefore ran f32 end to end. This port
has been accumulating WIDER than the oracle its own goldens came from.

The width was measured, not read

Reading upstream gives the tensor dtype, not the accumulator width. It was
settled by running torch 2.11.0 on a reduction engineered so the widths separate:
27 taps over a uniform input with weights [+1e8, 0.1 x 25, -1e8], where half an
ulp of 1e8 is 4.0 and the small taps sum to 2.5, so every partial sum rounds back
to 1e8 in any f32 order.

what	result
`F.conv3d`, f32 tensors	0.0
`F.conv3d`, bf16 tensors, upstream's own dtype	0.0
`F.conv3d`, f64 tensors	2.500000014901161
this port before this change	2.5

That is what makes the new gate non-circular: the value it asserts is torch's
answer, not a recording of the patched code. Every accumulator and elementwise
step on the data path narrows to f32. The epsilons, the once-per-block sqrt(C)
scalars and the TimestepEmbedding frequency table stay f64, each saying why at
its own site.

Width was not the whole mirror

Narrowing the width while keeping the naive serial summation order pushed
test_ltx2_tiling's non-causal untiled control to 5.00679e-06 against a 5e-06
tolerance, a real RED in the full gate. The fix is the ORDER, and it is a
closer mirror rather than a looser one: CausalConv3d now keeps one partial sum
per input channel, which is how torch's blocked-GEMM f32 convolution sums, and
which is why torch.sum returns 2.0999999 on the same reduction where a naive
serial f32 sum returns 0.0. No tolerance was touched.

The numerics did not move

Measured by zeroing kLtx2GoldenTol in both suites to read the values, then
restoring both.

golden arm	f64 (before)	naive f32	blocked f32 (shipped)	tol
Conv video decoder	1.40071e-06	4.12762e-06	1.72853e-06	5e-06
non-causal Conv video decoder	1.81794e-06	3.51667e-06	2.08616e-06	5e-06
tiled decode, untiled control A	2.08616e-06	5.00679e-06 FAIL	2.74181e-06	5e-06
tiled decode, untiled control B	2.62260e-06	(same case)	2.80142e-06	5e-06
norm_eps-binding video decoder	9.05246e-07	1.16974e-06	1.54972e-06	5e-06
cropped video encoder	4.17233e-07	8.94070e-07	4.76837e-07	5e-06
video encoder (`*_res`)	4.17233e-07	8.94070e-07	4.76837e-07	5e-06
causal-arm video encoder	2.98023e-07	3.83705e-07	4.17233e-07	5e-06
video encoder (strided convs)	5.96046e-07	5.96046e-07	8.34465e-07	5e-06
all 13 audio arms	unchanged	unchanged	unchanged	untouched

Against the f64 arm the shipped arm is 1.07x to 1.31x on every video golden. The
worst sits at 56% of tolerance, against the f64 arm's 52%.

And the existing goldens could not see the change at all — they pass on the
f64 arm and on the f32 arm alike. That is the point of #1008, and it is why this
ships its own instrument.

Mutations, each with three facts

mutation	built	exit	detected by
W1, conv accumulator widened to `double`	yes, 0 errors	1	the new width case, 1.03473 against 7
W2, `Linear3d` accumulator widened	yes, 0 errors	0	nothing, 40/40 pass
W3, `PixelNorm` `mean_sq` widened	yes, 0 errors	1	the Conv video decoder golden, incidentally
R, production call site deleted	yes, 0 errors	1	the new width case, 7 against 7

W2 is an honest gap and it is owed: nine sites are narrowed on upstream grounding
and review, not on a test.

Reachability, and the draft that passed while measuring nothing

The case enters through Ltx2VideoDecodeStreaming, the entry
src/vllm/multimodal/ltx2_video.cpp:3258 calls on the render path, reaching
Ltx2ConvVideoDecode at ltx2_video_vae_tiled.cpp:113. Deleting that call site
turns it RED.

It did not at first. The initial draft expected zero, which is what an f32
accumulator produces, and mutation R replaced the decode with a zero-filled
buffer, so the case PASSED while measuring nothing. conv_out.conv.bias = 7
moves the expectation off zero: f32 returns exactly 7, f64 returns 8.03473, a
stub returns 0. That first mutation attempt also failed to build (3 -Werror
errors) while a stale binary printed a plausible verdict, which is why every
mutation above reports whether it built.

NDHWC: filed, not half-built

The blocker is not Volume, whose At() has 16 call sites, but
MiniMaxH3GroupNorm3d (minimax_h3.h:756), whose signature hard-codes a
channel-major buffer and which MiniMax-H3's VAE CNN and the LTX-2 audio VAE also
call. And the win would not be the layout: upstream's channels_last_3d selects
a cuDNN kernel family, and this port has no device arm (#1007), so NDHWC is a
precondition for a SIMD or device arm rather than a speedup on a scalar loop.

No speed number

dgx.casa was unreachable for this row's whole duration, so the magnitude is
owed and unmeasured. One Ltx2ConvVideoDecode wall at a fixed size on an
idle host, same binary, f64 arm against f32 arm, settles it.

Gate

cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF, -j6, ctest -j4.

CONFIGURE_EXIT=0, BUILD_EXIT=0, : error: count 0, ctest -N 492,
CTEST_EXIT=0, 100% tests passed, 0 failed out of 492. No space left 0 and
BFD assertion 0, both greps positive-controlled against a synthetic line.
Load average 4.70 to 15.88 on a shared 20-core box; free disk 20-21 GiB of 447 GB.

An earlier run of the same gate failed test_async_llm (#294) at load average
22; it passes in isolation and contains zero ltx2 references, so this change
cannot reach it.

One deliberate omission

No .agents/issue-index.md row is appended for #1008. That row exists on PR
#1018, which filed the issue and is unmerged. .gitattributes:7 sets
merge=union on that file and scripts/check-agent-record.py:1437-1442 refuses
a duplicate issue number with "duplicate is what two branches appending the same
issue look like". Appending it here would turn main red for every branch the
moment #1018 merges, which is what a duplicate #995 row just did. The link lives
in the spec and in this body; the index link arrives with #1018.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

…s is f32 even in bf16 (#1008) Lever 2 of the LTX25-DECODE-SPEED investigation (#1006, PR #1018) split into the dtype half. The parent row filed #1008 on the observation that the LTX-2.5 conv video VAE decode declares `double acc` at 8 sites and casts 29 operands to double, where no reference stores f64 anywhere on the path. Reading the references gives the tensor dtype, not the accumulator width, so this spec settles the width by running torch 2.11.0 on a reduction engineered to separate the two: 27 taps over a uniform input with weights [+1e8, 0.1 x 25, -1e8], where every partial sum sits below half an ulp of 1e8 and an f32 accumulator lands on exactly zero in any summation order. F.conv3d returns 0.0 on f32 tensors AND on bf16 tensors, upstream's own configuration; the f64 arm returns 2.5, which is what this port returns today. That measurement is what makes the new gate non-circular: the value it asserts is torch's answer, not a recording of the patched code. It also bounds what may be gated, and the spec records the bound rather than overselling it -- torch.sum and torch.mean on the same input return 2.0999999, so torch's reductions are f32-wide but cascading where this port is naive serial. Width matches, order does not, so no adversarial gate is built through PixelNorm. The spec also records why the goldens cannot see any of this, from their own generator rather than by inheriting the claim: gen-ltx2-vae-goldens.py:223 casts every upstream parameter to np.float32, so the oracle that produced them ran f32 end to end. The useful corollary is that this port has been accumulating WIDER than the oracle its own goldens came from. The NDHWC half of the lever is filed, not half-built, and the spec names the blocker precisely: it is not `Volume`, whose At() has 16 call sites, but MiniMaxH3GroupNorm3d, whose signature hard-codes a channel-major buffer and which MiniMax-H3's VAE CNN and the LTX-2 audio VAE also call. No speed number is claimed. dgx.casa was unreachable throughout, so the magnitude is booked as owed and unmeasured with what would settle it. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

…es in f32 even at bf16 (#1008) The LTX-2.5 conv video VAE decode declared `double acc` at 8 sites and cast 29 operands to double. No reference accumulates in f64 anywhere on this path, and until now nothing here could report that: the goldens are vacuous on accumulator width by construction, because gen-ltx2-vae-goldens.py:223 casts every upstream parameter to np.float32 and the oracle therefore ran f32 end to end. This port has been accumulating WIDER than the oracle its own goldens came from. Reading upstream gives the tensor dtype, not the accumulator width, so the width was measured. On a reduction engineered so the two widths separate -- 27 taps over a uniform input with weights [+1e8, 0.1 x 25, -1e8], where half an ulp of 1e8 is 4.0 and the small taps sum to 2.5, so every partial sum rounds back to 1e8 in any f32 order -- F.conv3d returns 0.0 for f32 tensors AND for bf16 tensors, upstream's own dtype, while an f64 accumulator returns 2.5. Every accumulator and every elementwise step on the data path therefore narrows to f32. The epsilons, the once-per-block sqrt(C) scalars and the TimestepEmbedding frequency table stay f64 and each says why at its own site. WIDTH WAS NOT THE WHOLE MIRROR. Narrowing it while keeping the naive serial summation order pushed test_ltx2_tiling's non-causal untiled control to 5.00679e-06 against a 5e-06 tolerance, a real RED. The fix is the ORDER, and it is a closer mirror rather than a looser one: CausalConv3d now keeps one partial sum per input channel, which is how torch's blocked-GEMM f32 convolution sums, and which is why torch.sum returns 2.0999999 on the same reduction where a naive serial f32 sum returns 0.0. No tolerance was touched. With both fixed the numerics essentially do not move: against the f64 arm the shipped arm is 1.07x to 1.31x on every video golden, where the naive f32 arm was 1.9x to 2.9x. The Conv video decoder golden reads 1.40071e-06 (f64), 4.12762e-06 (naive f32), 1.72853e-06 (shipped). Measured by zeroing the tolerance in both suites to read the values, then restoring both. The new case enters through Ltx2VideoDecodeStreaming, the production entry the render path calls, and asserts an analytic consequence rather than a recorded vector. Its conv_out bias of 7 is load-bearing: the first draft expected zero, and the reachability mutation -- deleting the Ltx2ConvVideoDecode call at ltx2_video_vae_tiled.cpp:113 -- PASSED, because a decode that never ran returns zeros too. That first mutation attempt also failed to build while a stale binary printed a plausible verdict, which is why the spec records whether each mutation built. Gating is uneven and the spec says so rather than implying coverage: widening Linear3d back to double is caught by NOTHING, 40/40 cases pass, so nine sites are narrowed on upstream grounding and review rather than on a test, and that gap is owed. No speed number is claimed. dgx.casa was unreachable throughout, so the magnitude is owed and unmeasured. The NDHWC half of the lever is filed rather than half-built: the blocker is not `Volume`, whose At() has 16 call sites, but MiniMaxH3GroupNorm3d, whose signature hard-codes a channel-major buffer and which MiniMax-H3's VAE CNN and the LTX-2 audio VAE also call. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

…al tree Three `ltx2_video_vae.cpp` anchors in the spec were written against the base and went stale inside this pull request, because the change adds a dtype block to the file header and shifts everything below it. `Volume` moved 68-76 -> 85-93 and the phase L6 note 46-49 -> 63-66, both re-derived by searching for the claim's own text rather than by reading the cited span. The defect-site table in section 2 keeps its base anchors deliberately, since it locates the f64 sites as they stood before this change removed them, and it says so. Section 5's bypass-site list now says which basis it uses instead of leaving a reader to assume. The unchanged anchors were re-checked rather than trusted: ltx2_video_vae_tiled.cpp:113 still holds the sole Ltx2ConvVideoDecode call, and ltx2_video.cpp:3258 still holds the Ltx2VideoDecodeStreaming entry. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

…e the page said could not (#1088) (#1090) Closes #1088. `docs/USAGE.md` published 448x256 at 25 frames as "Measured NOT to complete", with the reason that its decode loses about 59 GB in 24 s. Two renders on `dgx.casa` on 16 to 17 August 2026 against `main` `0b0b8900f` completed that geometry in 3085 s and completed 704x448 at 25 frames in 4231 s. The page now records the newer envelope, and `docs/BENCHMARKS.md` no longer says the opposite of it in a cell a reader meets first. ## What was measured Container `vllmcpp-build:gb10`, `Release`, `VLLM_CPP_CUDA=ON`, arch `121a`, `TRITON=ON`, CUTLASS absent so FlashAttention-2 was not built, which is like for like with the earlier renders. `VLLM_CPP_CPU_THREADS=20`, NVFP4 transformer, no `--allow-unported`. `0b0b8900f` carries #1041 threaded decode, #1032 T2A and #1036 f32 decode accumulators. | Geometry | Result | Wall | |---|---|---| | 448x256 / 25 frames | completed | 3085 s | | 704x448 / 25 frames | completed | 4231 s | | 1024x576 / 25 frames | not attempted to completion, another session claimed the box | n/a | The ~59 GiB cliff did not recur under a 2 s memory guard that would have seen it: `MemAvailable` floors of 38.96 GiB over 1289 samples at 448x256 and 38.89 GiB over 1743 samples at 704x448, zero samples under 34 GiB on either, peak use of 80 of 119 GiB, and no reboot. The 704x448 artifact was verified rather than inferred from an exit code: 25/25 distinct frame md5s, 0 near-uniform and 0 near-black frames, adjacent-frame mean absolute difference 4.381 against a uniform-noise reference of 85.3 on the same shape, 0/24 zero-motion pairs, and audio at 48 kHz stereo, 1.010 s, -37.29 dBFS, 20/20 windows above threshold. The mp4 lives at `benchmarks/media/ltx25-704x448-25f-audio.mp4` on the render host and is gitignored by `.gitignore:35`, so it is not committed here. ## What is not claimed One run per geometry on a contended shared box with no oracle on either side. Two points establish no scaling law. 704x448 is not published as a ceiling: the next rung up stopped because another session claimed the box, not because of memory or an envelope. The page says all of this in its own words. ## The 59 GB is kept, not deleted It is the reason the old row gave, so deleting it would remove the evidence the newer result is measured against. It stays attributed to its own run, which is rung F1 in `.agents/benchmark-record.md`: a prompt-embeds render with no text tower that an armed watchdog ended at 13.77 GiB against an 18 GiB floor, rather than the engine failing. Attributing the fall is still #1014, and this change does not close it. ## The dominant cost moved off the decode `docs/USAGE.md` said most of a 320x192/25f render is spent in the host VAE decode. After #1041 threaded that decode, the dominant cost is a resolution-independent phase of about 1731 s, measured at 1731 s and 1732 s across two rungs whose voxel counts differ 2.75x, which is 57 to 66% of wall. That is #1087, which owns naming the phase. The sampler classified by CPU-time rate rather than by symbol, so what is measured is a duration and a scaling law and not a function, and the page says so. ## Files | Record | Edit | |---|---| | `docs/USAGE.md` | envelope table rows, the paragraph under it, the bounded-by paragraph, and the #1009 paragraph's stale "has not been re-measured" clause | | `docs/BENCHMARKS.md` | the `LTX-2.5 axes` row, edited in place as two table cells, 208 and 214 characters against `MAX_CELL_CHARS = 220`, so no prose paragraph is added to a page sitting at 35 of 35 | | `.agents/specs/ltx25-resolution-envelope.md` | new section 4.1 recording what superseded section 4, and the `## Owed` bullet that section 4 wrote | | `.agents/issue-index.md` | one row appended for #1088, zero rows edited, zero removed | ## Evidence Records only. No `src/`, `include/` or `tests/` change, so no build was run and none is claimed. Key-by-key proof, taking `HEAD`'s version of each file and reapplying the scoped edit: | Record | Keys in base | Keys now | Unrelated keys byte-identical | Changed | Added | Removed | |---|---|---|---|---|---|---| | `docs/USAGE.md` | 205 | 206 | 203 of 203 | `**Measured to complete on one GB10**` | `Largest size tried`, `Superseded, kept for the record` | `Measured NOT to complete` | | `docs/BENCHMARKS.md` | 190 | 190 | 189 of 189 | `LTX-2.5 axes` | none | none | Issue index, the three verifications the append-only rule needs: the base file is a byte-identical prefix of the new one, the addition is exactly one line whose sha256 is `65933626d961a41b…`, and the file has 290 rows against 290 unique issue ids. The union driver was never allowed to resolve anything: the file was rebuilt as base bytes plus the row. Checkers, each with a red control observed on the same tree before the green was believed: | Checker | Result | Armed control | |---|---|---| | `check-doc-checkpoint.py --staged` and `--commit cedb85e` | 0 | `--commit b5618b3` exits 1, "changed user_usage but did not update docs/USAGE.md" | | `check-public-doc-tables.py` | 0 | padding the new cell past 220 characters exits 1 at line 487, "table cell of 333 chars exceeds 220" | | `check-issue-index-append-only.py --base origin/main` | 0 | committing a deletion of the `#168` row exits 1, "this range removes or edits lines" | | `check-agent-record.py` | 0 | replacing the new row's owning row with a dash exits 1, "34 rows name no owner, above the recorded 33" | | `check-commit-style.py --range origin/main..HEAD` | 0 | an empty commit whose subject ends in a period exits 1 | | `check-commit-trailers.py --range origin/main..HEAD` | 0 | an empty commit with no trailer block exits 1 on three lines | | `check-pr-size.py --base origin/main --head HEAD` | 0 | n/a, no control run | Every tree mutation was restored and the restored file re-hashed to the pre-mutation sha256 before the next step. The key proof itself was seen red first, on an expectation that omitted the one key the change does edit in place, so its green is not a tautology. `scripts/agent-preflight.sh --staged` and `scripts/agent-ready.py` both report `All gates green` on `21544efd9`. `agent-ready` then exits 1 only on `expected exactly one live PR for row/LTX25-ENVELOPE-RECORD; found 0`, which this pull request is. `origin/main` advanced twice during this work, to `e9dfa6319` and then `9143196c7`. Both were merged in and every checker re-run afterwards; the second merge is the merge commit on this branch, and its message carries the trailer block because the range gate caught that it did not. ## What could not be verified The first `scripts/agent-preflight.sh` run exited 1 on `test_cpu_x86_llamacpp_floor`, on the unmodified tree before any edit in this branch. Its own output names the cause: `load=120.50`, so the harness discarded the contended leg and returned `NO_QUIET_WINDOW` (4) where the case expects `GIVING_UP` (2). That is #618. It passed on the later runs once the box quieted, so this branch has no evidence of that case being sound, only of it being load-dependent as #618 already says. The renders themselves were performed by another session and are reported here from its results. This branch did not run them, holds no GPU, and did not rebuild anything. `.agents/specs/ltx25-decode-speed.md` and `.agents/benchmark-record.md` also discuss the 448x256 rung. Neither is edited here: the decode-speed spec already records that the "inside the decode" half of the old sentence is unsupported, and the benchmark record is an append-only log of what each run observed, which stays true of the run it describes. Reconciling the investigation spec against the new rungs belongs to #1087, which owns the phase. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

mudler added 3 commits August 16, 2026 14:27

localai-bot merged commit d1b0ea3 into main Aug 16, 2026
19 of 25 checks passed

Navigation Menu

fix(LTX25-DECODE-DTYPE): the decode computed in f64, and torch computes in f32 even at bf16 (#1008) - #1036

fix(LTX25-DECODE-DTYPE): the decode computed in f64, and torch computes in f32 even at bf16 (#1008)#1036
localai-bot merged 3 commits into
mainfrom
row/LTX25-DECODE-DTYPE

localai-bot commented Aug 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

localai-bot commented Aug 16, 2026

The width was measured, not read

Width was not the whole mirror

The numerics did not move

Mutations, each with three facts

Reachability, and the draft that passed while measuring nothing

NDHWC: filed, not half-built

No speed number

Gate

One deliberate omission

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants