spec(LTX25-DECODE-SPEED): the decode has no device arm, and the 60 GiB was never in it - #1018
Closed
localai-bot wants to merge 6 commits into
Closed
spec(LTX25-DECODE-SPEED): the decode has no device arm, and the 60 GiB was never in it#1018localai-bot wants to merge 6 commits into
localai-bot wants to merge 6 commits into
Conversation
…B was never in it Two records sent this row after one problem, and there are two. The video VAE decode takes 2681 s at 448x256/25f, and a render loses ~59 GiB. They are unrelated, and treating them as one is what kept both open. The decode is the CPU reference arm, and production executes it. The file says in its own header that no throughput number should be taken from it (ltx2_video_vae.cpp:46-49), and it is nonetheless what every render runs. Its convolution accumulates in double at 8 sites with 29 static_cast<double>; no oracle accumulates in f64 anywhere on this path. vt::OpId::kLtx2 turns out to cover the DiT only, so there is no device path to wire -- the arm does not exist. One decode is ~7.25 TFLOP over 42 convs, which at 2681 s is 2.7 GFLOP/s: the decode is not algorithmically different from upstream, it is 7.25 TFLOP run one scalar f64 FMA at a time on one of twenty cores. A completed 49-frame render's own log, never read as evidence before, holds load 1.0-1.3 on a 20-core box for 89% of its wall. The 59 GiB is excluded from the decode twice: 361.72 MiB measured, and a 9.649 GiB ceiling computed by assuming the decode never frees anything, still 6x short. So memory_efficient_decode.py cannot be the cause, which closes the hypothesis the dispatch was built on rather than carrying it. The hypothesis that replaces it is model residency -- ~68 GiB documented as staged before any decode instruction, ~72 GiB observed acquired in ten minutes and held flat for two hours -- and it is filed as a hypothesis with the one-clock trace that settles it, not as a finding. Two oracle records are wrong and are corrected: diffusers implements LTX-2.5, both decode arms, and is already admitted and already pinned, while the campaign records that no admitted oracle carries 2.5; and SGLang-Omni implements nothing in this class. Neither correction closes #655, because ltx_core is what every LTX-2.5 correctness gate already runs against, and it is installed on no gate host. No product code. Each lever is filed for its own row, its own spec and its own fresh review, because bundling a new CUDA op with a dtype change no golden can see would put the second behind the first's review. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…ng one, and the plateau refutes it The first draft attributed the 59 GiB to model residency held across the decode: ~68 GiB is documented as staged and resident, a 49-frame render acquires ~72 GiB in ten minutes and holds it, and the sum lands near 60. Reading benchmark-record.md:21144-21160 refutes it. At 448x256/25f, on a prompt-embeds run with no text tower on the machine at all, both denoise phases finish and drain, MemAvailable is FLAT at 75.2 GiB through all of it, and only then does it fall 59 GB in 24 seconds. A load-time total cannot explain a fall that starts after a flat plateau -- the residency was already paid at the plateau. The correction is recorded rather than deleted, because summing the resident objects to "about 60 GiB" is the answer the next reader will also reach. What replaces it is a mechanism rather than a culprit. Backend::Alloc on CUDA is a raw cudaMalloc (cuda_backend.cu:77-81), and on GB10's unified pool that does not appear in VmRSS. So "flat RSS while MemAvailable fell 60 GiB" is not evidence that nothing allocated; it is the signature of a device-class allocation, and it is why every RSS sampler aimed at this came back empty. The record already knew the instrument was weak -- nvidia-smi --query-gpu=memory.used returns [N/A] on this box -- but --query-compute-apps=used_memory does work and was never used. Three hypotheses are now ranked, including that the fall may not be our process at all, and the queued probe carries the columns all three prior records lacked. Two residency defects turned up while establishing that, neither of which is the fall, both filed: Ltx2WidenDitToF32 appends the f32 buffer and never drops the bf16 original, so a host-arm load holds ~105.9 GiB of DiT weights on a 119 GiB box (#1015); and no ltx2 loader calls MaybeReleaseSourcePages, against 15 other files under src/vllm that do (#1016). USAGE's "inside the decode" is now recorded as half supported rather than wrong: the fall is on the decode side of the boundary, and the decode does not allocate it. Those are different claims and the doc collapses them. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…ins 0.0% of a 43.5 GiB fall Rung 1 took the GPU lock at 10:59:57Z after queuing behind a llama-imatrix holder and two waiters. It measured the mechanism section 4.3 could only argue. Over the DiT staging phase the process's Anonymous grew by 0.01 GiB against a 43.50 GiB MemAvailable fall. The device compute-app counter grew 30.98 GiB and file-backed RSS grew 10.83 GiB; together they are 96% of it. So an RSS- or Anonymous-based sampler is structurally blind here, which is why every earlier attempt to attribute this came back empty -- and nvidia-smi's per-process compute-app query, which does work on GB10 unlike --query-gpu=memory.used, was available the whole time. That promotes #1016 from "not excluded" to measured: the 10.83 GiB of file-backed RSS is the mmap source pages no ltx2 loader releases, growing in lockstep with the device copy on the same unified pool. The rung failed where nobody was looking. With the text tower on the path, 448x256/25f runs out of headroom during LOAD -- 35.54 GiB device plus 44.77 GiB host RSS is ~80 GiB before a single denoise step. The benchmark-record trace this spec quotes was a prompt-embeds run with no tower, which is why it had 75.2 GiB free to plateau at. Two different failures at one resolution, and either trace read alone invites conflating them. A lever no record names falls out of the same phase and is filed as #1021: staging is 450 s at ~52 MiB/s with the GPU idle in 164 of 192 samples and 0.15 of one core busy -- ~3,504 serial cudaMalloc + copy + Synchronize round trips at ltx2_loader.cpp:738-756. It is neither GPU-bound nor CPU-bound. The staged plateau lands within 1% of the loader contract's own figure, so the model that predicts it is validated. The instrument's own defect is recorded rather than quietly fixed: the sampler's pgrep -f matched the sudo/docker wrapper instead of the workload, so its per-PID columns came back blank -- and a blank Anonymous column reads as "the process allocated nothing". A side-car using pgrep -x supplied every per-PID figure here. Rung 2, the size that completes, has not returned; section 1.3 says so rather than borrowing rung 1's confidence. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…der, not just the decode Rung 2 measured 320x192/25f with --device cuda. Across 347 per-PID samples at 2 s, GPU utilization never exceeded 2% and was exactly 0 in 321 of them -- and every one of the 26 non-zero samples falls inside the DiT staging window, where it is copy traffic rather than compute. From t=251 s onward, for over 17 minutes, it is 0 in every sample while the compute-app footprint sits flat at 35.54 GiB and the process holds exactly 1.00 core of 20. No frame written. So the render stages 35.54 GiB of transformer weights onto a GPU that then does no work. Staging demonstrably took the device path -- Anonymous stayed at 0.01 GiB through it, so no f32 widening happened and on_device is true. That makes USAGE's "most of a render is spent single-threaded in the host VAE decode at 0% GPU" an understatement: the 0% GPU is not a property of the decode phase, it is a property of the render after load. Filed as #1024, and it bounds every other lever, because if the denoise is also on the host then #1007 alone does not close the render. What it leaves is a disjunction rather than an answer, and it is written as one: either the denoise is not taking Ltx2DitForwardDevice, or it is and something before it burns 17 minutes of single-threaded host time. The render emits nothing that timestamps a phase boundary, so this cannot currently be split -- which is precisely why #1010 was ranked as a precondition rather than a nicety. The instrument caveat is stated before the conclusion leans on it: there is no positive control that utilization.gpu reads high for a real kernel on GB10. It is live -- it moved 0-3% during staging -- but live is not correctly scaled, and this box already returns [N/A] for --query-gpu=memory.used. The control is owed and is one command under the lock. The CPU column needs no such control and carries the claim on its own: a full core of process time, a flat device counter, and no output are together only consistent with the host doing the arithmetic. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…, then exited 1 and the box went away The rung that was expected to complete did not. EXIT=1, 0 frames, 2407 s, 1082 sampler rows, no watchdog armed -- so the non-zero exit is the engine's own. The gate host then became unreachable minutes after the probe finished and has not answered since, so run.log was never read. The failure reason is REMOTE_UNVERIFIED and is recorded as such rather than guessed; MemAvailable had risen to 66.51 GiB shortly before the exit, which does not fit memory exhaustion at that moment. What the 1082 rows established does not depend on the exit reason. After load the render has at least four distinct compute regimes -- a multi-threaded host text encode at 2.9 cores, then ONE thread for about 26 minutes at exactly 1.000 core in every window measured, then a 15.1-core burst, then a ~30 GiB release -- and none of them uses the GPU. Two of those change earlier conclusions. The engine DOES release the text tower: the 30 GiB drop is it, so lever 5's "held across the decode" framing is wrong for this configuration too and #1014 should be read against it. And the 1.000-core window is the largest block of the run, roughly 26 of 40 minutes, but it cannot yet be NAMED as the VAE decode, because no frame was written and nothing in the tree timestamps a phase boundary. Naming it needs #1010, not another probe -- which is the second time this row has hit that wall and is why #1010 is ranked as a precondition rather than a nicety. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
localai-bot
added a commit
that referenced
this pull request
Aug 16, 2026
…es in f32 even at bf16 (#1008) (#1036) Lever 2 of the `LTX25-DECODE-SPEED` investigation (#1006, PR #1018), dtype half. Closes #1008. The LTX-2.5 conv video VAE decode declared `double acc` at 8 sites and cast 29 operands to double. No reference accumulates in f64 anywhere on this path, and until now nothing here could report that: the goldens are vacuous on accumulator width by construction, because `gen-ltx2-vae-goldens.py:223` casts every upstream parameter to `np.float32` and the oracle therefore ran f32 end to end. This port has been accumulating WIDER than the oracle its own goldens came from. ## The width was measured, not read Reading upstream gives the tensor dtype, not the accumulator width. It was settled by running torch 2.11.0 on a reduction engineered so the widths separate: 27 taps over a uniform input with weights `[+1e8, 0.1 x 25, -1e8]`, where half an ulp of 1e8 is 4.0 and the small taps sum to 2.5, so every partial sum rounds back to 1e8 in any f32 order. | what | result | |---|---| | `F.conv3d`, f32 tensors | **0.0** | | `F.conv3d`, bf16 tensors, upstream's own dtype | **0.0** | | `F.conv3d`, f64 tensors | 2.500000014901161 | | this port before this change | 2.5 | That is what makes the new gate non-circular: the value it asserts is torch's answer, not a recording of the patched code. Every accumulator and elementwise step on the data path narrows to f32. The epsilons, the once-per-block `sqrt(C)` scalars and the `TimestepEmbedding` frequency table stay f64, each saying why at its own site. ## Width was not the whole mirror Narrowing the width while keeping the naive serial summation order pushed `test_ltx2_tiling`'s non-causal untiled control to **5.00679e-06 against a 5e-06 tolerance**, a real RED in the full gate. The fix is the ORDER, and it is a closer mirror rather than a looser one: `CausalConv3d` now keeps one partial sum per input channel, which is how torch's blocked-GEMM f32 convolution sums, and which is why `torch.sum` returns 2.0999999 on the same reduction where a naive serial f32 sum returns 0.0. No tolerance was touched. ## The numerics did not move Measured by zeroing `kLtx2GoldenTol` in both suites to read the values, then restoring both. | golden arm | f64 (before) | naive f32 | blocked f32 (shipped) | tol | |---|---|---|---|---| | Conv video decoder | 1.40071e-06 | 4.12762e-06 | **1.72853e-06** | 5e-06 | | non-causal Conv video decoder | 1.81794e-06 | 3.51667e-06 | 2.08616e-06 | 5e-06 | | tiled decode, untiled control A | 2.08616e-06 | **5.00679e-06 FAIL** | 2.74181e-06 | 5e-06 | | tiled decode, untiled control B | 2.62260e-06 | (same case) | 2.80142e-06 | 5e-06 | | norm_eps-binding video decoder | 9.05246e-07 | 1.16974e-06 | 1.54972e-06 | 5e-06 | | cropped video encoder | 4.17233e-07 | 8.94070e-07 | 4.76837e-07 | 5e-06 | | video encoder (`*_res`) | 4.17233e-07 | 8.94070e-07 | 4.76837e-07 | 5e-06 | | causal-arm video encoder | 2.98023e-07 | 3.83705e-07 | 4.17233e-07 | 5e-06 | | video encoder (strided convs) | 5.96046e-07 | 5.96046e-07 | 8.34465e-07 | 5e-06 | | all 13 audio arms | unchanged | unchanged | unchanged | untouched | Against the f64 arm the shipped arm is 1.07x to 1.31x on every video golden. The worst sits at 56% of tolerance, against the f64 arm's 52%. **And the existing goldens could not see the change at all** — they pass on the f64 arm and on the f32 arm alike. That is the point of #1008, and it is why this ships its own instrument. ## Mutations, each with three facts | mutation | built | exit | detected by | |---|---|---|---| | W1, conv accumulator widened to `double` | yes, 0 errors | 1 | the new width case, 1.03473 against 7 | | W2, `Linear3d` accumulator widened | yes, 0 errors | **0** | **nothing, 40/40 pass** | | W3, `PixelNorm` `mean_sq` widened | yes, 0 errors | 1 | the Conv video decoder golden, incidentally | | R, production call site deleted | yes, 0 errors | 1 | the new width case, 7 against 7 | W2 is an honest gap and it is owed: nine sites are narrowed on upstream grounding and review, not on a test. ## Reachability, and the draft that passed while measuring nothing The case enters through `Ltx2VideoDecodeStreaming`, the entry `src/vllm/multimodal/ltx2_video.cpp:3258` calls on the render path, reaching `Ltx2ConvVideoDecode` at `ltx2_video_vae_tiled.cpp:113`. Deleting that call site turns it RED. It did not at first. The initial draft expected **zero**, which is what an f32 accumulator produces, and mutation R replaced the decode with a zero-filled buffer, so the case PASSED while measuring nothing. `conv_out.conv.bias = 7` moves the expectation off zero: f32 returns exactly 7, f64 returns 8.03473, a stub returns 0. That first mutation attempt also failed to build (3 `-Werror` errors) while a stale binary printed a plausible verdict, which is why every mutation above reports whether it built. ## NDHWC: filed, not half-built The blocker is not `Volume`, whose `At()` has 16 call sites, but `MiniMaxH3GroupNorm3d` (`minimax_h3.h:756`), whose signature hard-codes a channel-major buffer and which MiniMax-H3's VAE CNN and the LTX-2 audio VAE also call. And the win would not be the layout: upstream's `channels_last_3d` selects a cuDNN kernel family, and this port has no device arm (#1007), so NDHWC is a precondition for a SIMD or device arm rather than a speedup on a scalar loop. ## No speed number `dgx.casa` was unreachable for this row's whole duration, so the magnitude is **owed and unmeasured**. One `Ltx2ConvVideoDecode` wall at a fixed size on an idle host, same binary, f64 arm against f32 arm, settles it. ## Gate `cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF`, `-j6`, `ctest -j4`. `CONFIGURE_EXIT=0`, `BUILD_EXIT=0`, `: error:` count **0**, `ctest -N` **492**, `CTEST_EXIT=0`, **100% tests passed, 0 failed out of 492**. `No space left` 0 and `BFD assertion` 0, both greps positive-controlled against a synthetic line. Load average 4.70 to 15.88 on a shared 20-core box; free disk 20-21 GiB of 447 GB. An earlier run of the same gate failed `test_async_llm` (#294) at load average 22; it passes in isolation and contains zero `ltx2` references, so this change cannot reach it. ## One deliberate omission **No `.agents/issue-index.md` row is appended for #1008.** That row exists on PR #1018, which filed the issue and is unmerged. `.gitattributes:7` sets `merge=union` on that file and `scripts/check-agent-record.py:1437-1442` refuses a duplicate issue number with "duplicate is what two branches appending the same issue look like". Appending it here would turn `main` red for every branch the moment #1018 merges, which is what a duplicate #995 row just did. The link lives in the spec and in this body; the index link arrives with #1018. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
Collaborator
Author
|
Superseded by #1038. #1018's branch carried a merge commit #1038 is the same content rebased onto The |
localai-bot
pushed a commit
that referenced
this pull request
Aug 16, 2026
…nd the one that would have made a later agent weaken the test (#1044) Review of [PR #1041](#1041) returned PASS with five non-blocking findings. Each was re-verified before it was repaired, because a reviewer's finding is a hypothesis; none of the five was rejected. **F1 — the T1/T2/T3 gap now has an owner.** §8.6 measured that reverting any single one of the three dispatch sites is detected by nothing, and argued it in prose only. It is now [#1044](#1044), an entry in §7's `## Owed` table, and an index row owned by `LTX25-DECODE-THREADS`. The issue carries the closing test the reviewer supplied: a per-dispatch `Threadpool::RunCount()` and an EXACT expected count rather than `> 0`, plus a `res_x_y` fixture because `Linear3d` is unreachable with `decoder_blocks` empty. It is NOT implemented here — a new gate needs its own red-before evidence and its own review. The index row names the owning row rather than leaning on `## Owed`, and that is deliberate: `owed_issues()` in `scripts/check-agent-record.py` splits on a bare `\n## Owed`, this spec's heading is `## 7. Owed`, so nothing listed there is visible to the unowned ratchet. Measured: the unowned count is 33 before and after, against `UNOWNED_HIGH_WATER = 33`. **F2 — the public records carried three significant figures a 21-23% spread does not support.** `9.14x`/`9.67x` appeared bare in `FEATURES.md` and `USAGE.md` while the spec disclosed the load and the spread; the projection is what lost them. Both now carry `~9x at 16-20 workers` with the conditions, and the bare `9.67x at c=128` — n=3, no min/median/max, same contended box — is gone from `FEATURES.md` rather than restated. `MAX_CELL_CHARS = 220` binds and the `BENCHMARKS.md` LTX-2.5 cell sat at exactly 220: it is now 212, and the `FEATURES.md` cell 210 -> 204, both measured with the checker's own parser. **F3 — the #1009 index row was cited at `:275` on PR #1018.** #1018 is CLOSED and superseded by [#1038](#1038); the row lives at line **279** on `row/LTX25-DECODE-SPEED-R2`. The decision not to append a second #1009 row is correct and unchanged — under `merge=union` a duplicate reds `main` for every branch the moment #1038 lands. Only the citation moved, here and in the pull request body, along with the spec's two other pointers at the closed pull request. **F4 — the evidence SHA did not resolve.** §8 cited `dac85969c`, which is not an ancestor of the head (`git merge-base --is-ancestor` exits 1) and would not exist in a fresh clone. §8 now cites `d653f7319` and states why the measurement transfers: `dac85969c:src` and `d653f7319:src` are both `7444ffa171b0c2868c505b5b9ea1113fa39c5477`, both `:tests` are `f0e5eac268119e9fe94da478c50e2d668a2e64b3`, and the diff between them touches only the spec and three `docs/` files. **F5 — the determinism test's stated reason was arithmetically false, and that is the finding that mattered.** The comment said 3 and 5 were chosen because they "do not divide the row counts". Both conv row counts are 360 (`conv_in`, 24*3*5) and 15 (`conv_out`, 1*3*5), and 3 and 5 divide each of them. The choice works for a different reason: `nchunk` derives from `nth * 4`, so at `nr = 360` the stride is 45 at 2 workers, 30 at 3, 18 at 5 and 12 at 8 — four DIFFERENT partitions, which is the property the `memcmp` needs. Verified by replicating `cpu_threadpool.cpp:428-443` rather than by reading it. A later agent "fixing" the row counts to satisfy the stated rationale would have weakened the test while believing it was strengthening it, so the comment now says so outright. **Out of scope and untouched:** the parallelisation, the three dispatch sites, `kLtx2GoldenTol` (5e-06 on both arms), the new cases' assertions, and the `## Owed` ratchet beyond F1's entry. Nothing was re-measured: `dgx.casa` is down and the A/B harness is deliberately not in the tree, so every wall-clock figure stands as the implementer recorded it. Gate at this tree: `CONFIGURE_EXIT=0`, `BUILD_EXIT=0`, `: error:` count 0 on a full 1449-target build, `ctest -N` 492, `CTEST_EXIT=0`, **100% tests passed, 0 tests failed out of 492** in 165.47 s, the same two pre-existing skips. `No space left` 0 and `BFD`/internal-error 0 across both logs, each grep positive-controlled against a synthetic file carrying the real message forms (2, 1 and 2 hits there). One-minute load 10 to 26; free disk 40 GiB falling to 21 GiB. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
localai-bot
added a commit
that referenced
this pull request
Aug 16, 2026
…d one (#1009) (#1041) Lever 3 of the `LTX25-DECODE-SPEED` investigation (#1006, PR #1038 — #1018 was the earlier pull request and is now closed). Closes #1009. `ParallelForRows` (`src/vt/cpu/cpu_threadpool.cpp:413`) is synchronous and 10+ CPU kernels in this tree dispatch through it. Zero of them were in the LTX-2.5 conv video VAE decode, whose 42 convolutions carry ~7.25 TFLOP at 448x256/25f and ran on one core of twenty. Three sites now dispatch: `CausalConv3d`'s output nest, its padding gather, and `Linear3d`. ## The axis is the whole risk, so it is argued at the site The sibling dtype row (#1008, `d1b0ea3a8`) had to change this convolution's summation **order** to a blocked one to stay inside a 5e-06 tolerance. Parallelism is the second thing that can change a summation order. The partition is the output line `(oc, ti, hi)`, `out.w` contiguous elements. `Volume::At(oc, ti, hi, wi)` is `((oc*t + ti)*h + hi)*w + wi`, so row `r` is exactly `[r*out.w, (r+1)*out.w)` of `out.data` — no element is written twice — and the entire `ci * kernel^3` reduction stays inside one output element's body in the blocked order #1008 shipped. A worker therefore executes the serial arm's instruction sequence, in the serial arm's order, on the serial arm's values, for every element it owns. The result cannot depend on the worker count **or on which worker stole which chunk**, and the second half matters: `ParallelForRows` steals through an atomic cursor, so the row-to-thread assignment is genuinely non-deterministic run to run. Splitting the reduction axis `ic` into per-thread partials would also be a legal convolution. It is rejected in the comment at the site, because it would make the summation order a function of the thread count. ## The numerics did not move at all Both suites were rebuilt with `kLtx2GoldenTol` set to `0.0` before the change and again after, so every golden reports its `max|diff|` rather than its verdict. **All 34 recorded margins — 23 in `test_ltx2_vae`, 11 in `test_ltx2_tiling` — came back byte-for-byte identical**, compared by diffing the two sorted lists rather than by eye: `VAE_MARGINS_IDENTICAL (23 values)`, `TILING_MARGINS_IDENTICAL (11 values)`. | golden arm | before (serial) | after (20-thread global pool) | tol | |---|---|---|---| | Conv video decoder | 1.72853e-06 | 1.72853e-06 | 5e-06 | | non-causal Conv video decoder | 2.08616e-06 | 2.08616e-06 | 5e-06 | | norm_eps-binding video decoder | 1.54972e-06 | 1.54972e-06 | 5e-06 | | tiled decode, untiled control A | 2.74181e-06 | 2.74181e-06 | 5e-06 | | tiled decode, untiled control B | 2.80142e-06 | 2.80142e-06 | 5e-06 | | every other arm in both suites | unchanged | unchanged | — | Those before-values are also the ones `ltx25-decode-dtype.md` §8.1 recorded on its own host, which is an independent check that this box reproduces the sibling row. No tolerance was touched. That table is itself a threading gate: the suite runs on the global pool, `hardware_concurrency` wide, so every LTX-2.5 video golden after this change executes on 20 workers, and the "Conv video decoder" fixture carries a `res_x_y` block so `Linear3d` and `conv_shortcut` are on that path too. ## Two cases, because one of them measures nothing on its own A thread-count A/B is green on a serial implementation. Shipping only that would have been a test that passes while measuring nothing. * **"the decode DISPATCHES its convolutions to the CPU threadpool"** reads the pool's public work-stealing cursor through `ChunkAdd(0)`, which is a non-mutating read. A fresh pool reads 0; a pool that has run a partitioned dispatch reads at least `nth`. Asserting 0 *before* the decode is the instrument's own positive control. **Before this change it fails `CHECK( 0 > 0 )`.** * **"the decode is BIT-IDENTICAL across thread counts"** decodes the same latent at 1, 2, 3, 5 and 8 workers and `memcmp`s every arm against the 1-worker one, which short-circuits to the pre-change serial path. **3 and 5 are there because `nchunk` derives from `nth * 4`, not from `nth`** — 45/30/18/12 are the chunk strides at this fixture's 360 `conv_in` output lines, four different partitions of the same output. It is NOT that 3 and 5 fail to divide the row counts: 360 and 15 are both divisible by each. That was the comment's original claim and it was false; see the repairs below. Both enter through `Ltx2VideoDecodeStreaming` — what the render path calls at `src/vllm/multimodal/ltx2_video.cpp:3258` — and both assert an analytically derived value of exactly **7**, not a recorded one, because #1008 recorded that a zero-filled stub satisfies an expectation of zero. ## The CPU A/B Same binary, `VLLM_CPP_CPU_THREADS` the only variable, one decode through `Ltx2VideoDecodeStreaming`, 14 runs per count across an ascending and a descending sweep so an ordering drift shows as spread rather than hiding in a mean. | threads | runs | min s | median s | max s | spread | speedup | efficiency | |---|---|---|---|---|---|---|---| | 1 | 14 | 1.9859 | **2.0418** | 2.1172 | 6.4% | 1.00x | 100% | | 2 | 14 | 1.0266 | **1.0552** | 1.0843 | 5.5% | **1.93x** | 96.7% | | 4 | 14 | 0.5464 | **0.5555** | 0.5674 | 3.8% | **3.68x** | 91.9% | | 8 | 14 | 0.2920 | **0.3013** | 0.3072 | 5.0% | **6.78x** | 84.7% | | 16 | 14 | 0.2129 | **0.2232** | 0.2597 | 21.0% | **9.15x** | 57.2% | | 20 | 14 | 0.2024 | **0.2234** | 0.2534 | 22.8% | **9.14x** | 45.7% | A second shape at the checkpoint's real `base_channels` of 128: 5.1015 s at one thread against 0.5276 s at twenty, **9.67x**. That is the weakest number here — `n = 3` against the table's 14, no min/median/max, same contended box — so it corroborates the table's shape at a second channel width and is not independently a three-significant-figure result. The public records carry `~9x at 16-20 workers` with the conditions rather than either decimal. **The load it was taken at.** One-minute load average 4.03 to 6.77 on a box whose one-minute average had been between 2 and 94 the same day, with one non-agent process holding ~1.07 cores throughout. That process is part of why 16 and 20 spread 21-23% where everything at or below 8 spreads under 7%. **No ceiling is declared.** The implied serial fraction at 9.14x on 20 workers is 6.3%, which is the right order for `PixelNorm`, `Silu`, `ApplyAdaLn`, the residual add and `expand` — every one still serial, every one listed under `## Owed`. Memory bandwidth is the second candidate and is not separated here. **Determinism, proven a second time:** the output checksum was bit-identical across all **84** A/B decodes — six worker counts, two sweep directions, two shapes — on pseudo-random weights rather than the engineered fixture. ## What is NOT claimed No end-to-end render speedup, no ratio against any oracle, no composition figure with #1008. There is no GPU here, `dgx.casa` was unreachable throughout, and `ltx_core` is not installed. The harness shape is synthetic and says so; what generalises from it is the scaling, not the absolute wall. ## ThreadSanitizer, with the instrument controlled first `RelWithDebInfo` + `VLLM_CPP_SANITIZE=thread`: `test_ltx2_vae` 42/42, `test_ltx2_tiling` 10/10, `test_ltx2_video` 57/57, all `EXIT=0`, zero `WARNING: ThreadSanitizer`. Two instrument problems had to be settled before that meant anything. The binaries would not start at all — `FATAL: ThreadSanitizer: unexpected memory mapping`, `EXIT=66`, an ASLR-against-shadow-layout failure that a `&&` chain would have read as a race; `setarch x86_64 -R` fixes it. And a sanitizer that reports nothing is indistinguishable from one that is not instrumenting, so a deliberate unsynchronised write was compiled into `CausalConv3d`'s parallel body in the same lane: **87** `WARNING: ThreadSanitizer: data race`, `EXIT=66`, then reverted, rebuilt, and back to 0 and `EXIT=0`. ## Mutations, three facts each | mutation | numstat | built | exit | detected by | |---|---|---|---|---| | **T0** — all three dispatches reverted | 13/8 | yes, 0 errors | **1** | the dispatch case, `CHECK( 0 > 0 )` | | T1 — `CausalConv3d`'s output loop alone | 3/2 | yes, 0 errors | **0** | **nothing. 42/42 and 10/10 pass** | | T2 — the padding gather alone | 5/3 | yes, 0 errors | **0** | **nothing. 42/42 and 10/10 pass** | | T3 — `Linear3d` alone | 5/3 | yes, 0 errors | **0** | **nothing. 42/42 and 10/10 pass** | | D1 — chunk-dependent value, visible at 1 worker | 1/0 | yes, 0 errors | **1** | 10 cases + 2 tiling cases | | **D2** — the same defect INVISIBLE at 1 worker | 1/0 | yes, 0 errors | **1** | the bit-identity `memcmp`, on all four non-base arms | | **R** — production call site deleted | 17/2 | yes, 0 errors | **1** | the dispatch case on the cursor AND the value; the identity case's non-degeneracy `REQUIRE` | | T1, first attempt | 4/2 | **NO, 45 errors** | — | **nothing — a mutation that does not build establishes nothing** | **T1's failed first attempt is in the table on purpose.** One unbalanced brace closed the anonymous namespace early and produced 45 `-Werror` errors that read as unrelated `unused-function` complaints hundreds of lines away. The runner refused to draw a verdict rather than running a stale binary and printing a plausible 42/42. **T1, T2 and T3 are an honest gap and it is owed.** One work-stealing cursor is shared, so reverting any single site leaves the other two dispatching and the case reads non-zero. It gates *"at least one of the three sites dispatches"*, and T0 is what holds the conjunction. T3 additionally cannot be seen by that fixture at all, since `Linear3d` is only reached through a `res_x_y` block. What does bound each site is the golden table above — the "Conv video decoder" arm reaches all three at 20 workers and did not move — and the wall-clock, which is what a serial convolution would actually cost. **D1 is beside D2 because it is the weaker of the two.** D1 perturbs the first row of every chunk including the first, so the 1-worker arm moves too and the case fails on its value assertion before reaching the `memcmp`. D2 perturbs only chunks that do not start at row 0, which is invisible at one worker, so the `memcmp` across worker counts is the only thing that can report it. It does, on all four. ## Gate `cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF`, `-j6`, `ctest -j4`. Run twice, the second at this branch head so a green gate chains to the push. `CONFIGURE_EXIT=0`, `BUILD_EXIT=0`, `: error:` count **0**, `ctest -N` **492**, `CTEST_EXIT=0`, **100% tests passed, 0 tests failed out of 492** in both runs (308.99 s and 316.87 s). Two pre-existing skips, `test_modelopt_mixed_precision_checkpoint` and `test_voxtral_e2e`. `No space left` **0** and `BFD` internal-error/assertion **0** across every log, both greps positive-controlled against a synthetic file carrying the real message forms — 1 and 2 hits there, 0 in the real logs. `check-doc-checkpoint --commit` green on each of the three commits and armed (`b5618b305` exits 1). Load average 32-52 on the first run and **82-94** on the head run, on a shared 20-core box; none of the load-dependent suites flaked in either. Free disk 21-30 GiB of 447 GB; the 834 MiB sanitizer tree was removed after use. ## One deliberate omission **No `.agents/issue-index.md` row is appended for #1009.** That row already exists at `.agents/issue-index.md:279` on PR #1038, branch `row/LTX25-DECODE-SPEED-R2`, which filed the issue and is unmerged. #1018 was the pull request that carried it first; #1018 is closed and #1038 supersedes it. `.gitattributes` sets `merge=union` on that file and `scripts/check-agent-record.py` refuses a duplicate issue number, so a second copy here would turn `main` red for every branch the moment #1038 merges — which is what a duplicate #995 row did on 2026-08-16. The sibling dtype row made the same call for #1008. The link lives in the spec and in this body; the index link arrives with #1038. One index row IS appended by this branch, for the new issue #1044 below, and #1044 is not among the ids #1038 appends. ## The review, and the five findings repaired after it A fresh reviewer returned **PASS with no blocking findings**: the gate reran at 492/492, reduction safety was verified from `Volume::At`'s index arithmetic, determinism was proven by a mutation caught at all four worker counts, ThreadSanitizer was clean against an 84-race positive control, the keyed records were proven key by key, all nine anchors landed exactly, and a correctly-shaped zero-filled buffer fails both new cases, so the zero-stub trap is closed. Five non-blocking findings followed. Each was re-verified before it was repaired, because a finding is a hypothesis; **none of the five was rejected**. | # | Finding | Verified how | Repair | |---|---|---|---| | F1 | The T1/T2/T3 gap was argued in §8.6 prose with no issue and no `## Owed` entry | §7's table has no per-site row; `ParallelForRows` seeds ONE pool cursor (`cpu_threadpool.cpp:438`, advanced `:455`), so two surviving sites keep it non-zero | Filed [#1044](#1044), added it to §7 `## Owed` and to the issue index, owned by `LTX25-DECODE-THREADS` | | F2 | `FEATURES.md` and `USAGE.md` carried `9.14x`/`9.67x` bare, against a 21-23% spread | Cell widths measured with `check-public-doc-tables.py`'s own parser | `~9x at 16-20 workers` with the conditions; the bare `9.67x at c=128` is gone from `FEATURES.md` | | F3 | The #1009 index row was cited at `:275` on PR #1018 | #1018 is **CLOSED**; the row is at line **279** of the index on `row/LTX25-DECODE-SPEED-R2` (PR #1038) | Citation corrected here and in the spec, and the two other spec pointers at the closed pull request with it. The DECISION is unchanged | | F4 | §8 cited evidence at `dac85969c`, which does not resolve on the branch | `git merge-base --is-ancestor dac85969c HEAD` exits **1** | §8 cites `d653f7319` and states why the measurement transfers | | F5 | The determinism case's stated reason was arithmetically false | 360 and 15 are both divisible by 3 and by 5; the strides are 45/30/18/12 | The comment now states the real mechanism and says outright not to "fix" the row counts | **F1 is owed, not implemented.** #1044 carries the closing test the reviewer supplied — a per-dispatch `Threadpool::RunCount()` bumped in `Run()` and an EXACT expected count rather than `> 0`, plus a fixture carrying a `res_x_y` block because `Linear3d` is unreachable with `decoder_blocks` empty. A new gate needs its own red-before evidence and its own fresh review, so it is a row. **The index row names its owner rather than leaning on `## Owed`, deliberately.** `owed_issues()` in `scripts/check-agent-record.py` splits on a bare `\n## Owed` and this spec's heading is `## 7. Owed`, so nothing listed there is visible to the unowned ratchet. Measured: the unowned count is **33 before and after**, against `UNOWNED_HIGH_WATER = 33`. Four other specs have the same numbered heading (`ltx25-decode-dtype`, `ltx25-token-append`, `nemotron-h-a2q1-fp8-mamba`, `nemotron-h-a2q2-nvfp4-moe-lmhead`); that is a record observation this row does not repair. **Cell widths, since `MAX_CELL_CHARS = 220` binds and the `BENCHMARKS.md` cell sat at exactly 220.** `BENCHMARKS.md` LTX-2.5 axes 220 to **212**; `FEATURES.md` decode-threading 210 to **204**. Both measured with the checker's own `_table_rows`, and `check-public-doc-tables.py` exits 0. **Keyed records, proven key by key against the merge base.** `BENCHMARKS.md`: 179 unrelated keys byte-identical, only `LTX-2.5 axes` changed, none added or removed. `FEATURES.md`: 194 unrelated keys byte-identical. `USAGE.md`: 200 unrelated keys byte-identical, no table row touched. `.agents/issue-index.md`: `origin/main`'s version is a **byte-identical prefix**, exactly one appended line, and it is #1044. **Nothing was re-measured.** `dgx.casa` is down and the A/B harness is deliberately not in the tree, so every wall-clock figure above stands as the implementer recorded it. No end-to-end render speedup is claimed here either. **Gate after the repairs.** `CONFIGURE_EXIT=0`, `BUILD_EXIT=0`, `: error:` count **0** on a full 1449-target build, `ctest -N` **492**, `CTEST_EXIT=0`, **100% tests passed, 0 tests failed out of 492** in 165.47 s, the same two pre-existing skips. `No space left` **0** and `BFD`/internal-error **0** across both logs, each grep positive-controlled against a synthetic file carrying the real message forms (2, 1 and 2 hits there, 0 in the real logs). `check-doc-checkpoint --commit` green on all four branch commits and armed: `b5618b305` exits **1**. One-minute load 10 to 26; free disk 40 GiB falling to 21 GiB. ## Pushed with `--no-verify`, and why The `pre-push` hook refuses this branch on `check-public-doc-tables.py`: docs/BENCHMARKS.md has 36 prose paragraphs, over the 35 budget docs/FEATURES.md has 22 prose paragraphs, over the 21 budget **This branch did not cause it.** Matched-arm check: `origin/main` alone, in a detached worktree with no branch content, fails with the *identical* numbers. Bisected to `e34d71379` (#1054, an AppleClang capture fix that also added +5 lines to BENCHMARKS and +4 to FEATURES); `283c7e492` immediately before it exits 0. Filed as #1055. AGENTS.md: "Hooks are bypassable convenience, not evidence", and "a commit that needs an exception argues for it in its own message". This is that argument. The same checker run against this branch's own edits is clean -- it changes exactly one key in BENCHMARKS (`LTX-2.5 axes`) and adds exactly one in FEATURES (`LTX-2.5 Conv VAE decode threading`), with every unrelated key proven byte-identical to `origin/main`. ## Gate at the merged tree `CONFIGURE_EXIT=0`, `BUILD_EXIT=0`, `: error:` 0, 493 targets linked, `ctest -N` 495, **494 of 495 passed**. The one failure is `test_serve_low_tools`, which is **#428** ("the concurrency-cap assertion races the server-side counter and reads 3 under load") -- it passes 3/3 when re-run alone, and this branch touches no serve or tools file. Attribution verified against the issue that names the test, not assumed from a family. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two records sent this row after one problem, and there are two. The LTX-2.5 video VAE decode takes 2681 s at 448x256/25f, and a render loses ~59 GiB. They are unrelated, and treating them as one problem is what kept both open.
Spec and issues only. No product code. Each lever is filed for its own row, its own spec and its own fresh review, because bundling a new CUDA op with a dtype change no golden can see would put the second behind the first's review.
Why the decode is slow
It is the CPU reference arm, and production executes it.
src/vllm/multimodal/ltx2_video.cpp:3258reachesLtx2ConvVideoDecodeon the render path, while the file's own header says no throughput number should be taken from it (ltx2_video_vae.cpp:46-49). Its convolution accumulates indoubleat 8 sites across 29static_cast<double>; no oracle accumulates in f64 anywhere on this path. Andvt::OpId::kLtx2turns out to cover the DiT only, so there is no device path to wire — the arm does not exist.One decode is ~7.25 TFLOP over 42 convs, computed from the LTX-2.5 conv VAE config read out of the checkpoint header, and independently corroborated by an existing 3.5 TMAC derivation. At 2681 s that is 2.7 GFLOP/s. The decode is not algorithmically different from upstream; it is 7.25 TFLOP run one scalar f64 FMA at a time on one of twenty cores.
What the probe measured
Two rungs on
dgx.casa, queued behind$HOME/gpu.lock(never jumped: it waited out allama-imatrixholder and two waiters plus its own headroom guard).The 59 GiB mechanism is settled. Over the DiT staging phase the process's
Anonymousgrew 0.01 GiB against a 43.50 GiBMemAvailablefall — 0.0% of it — while the CUDA compute-app counter grew 30.98 GiB and file-backed RSS grew 10.83 GiB, together 96%. An RSS- orAnonymous-based sampler is structurally blind here, which is why every earlier attempt came back empty. That also promotes #1016 from "not excluded" to measured.The GPU is idle for the whole post-load render, not just the decode. 0% utilization in 321 of 347 samples, max 2%, and every non-zero sample is inside the staging window — on a run that staged 35.54 GiB onto that GPU with
--device cuda. Sodocs/USAGE.md:873-874understates it (#1024).Four post-load compute regimes, none using the GPU: a 2.9-core host text encode, then one thread for ~26 minutes at exactly 1.000 core, then a 15.1-core burst, then a ~30 GiB release. The largest block is the single-threaded one, and it cannot yet be named as the decode because nothing timestamps a phase boundary — which is why #1010 is ranked as a precondition.
Two answers this row first reached and then refuted, both recorded rather than deleted:
memory_efficient_decode.pyas the 59 GiB (a 9.649 GiB no-frees ceiling closes it), and model residency held across the decode (a flat 75.2 GiB plateau precedes the fall — and rung 2 shows the tower is released).Rung 2 exited 1 with 0 frames after 2407 s, and the host went unreachable before
run.logcould be read. That reason isREMOTE_UNVERIFIEDand stays so.Two oracle records corrected
diffusersimplements LTX-2.5, both decode arms (ltx2_diffusion_decoder.py:702, "introduced in LTX-2.5"), and is already admitted and already pinned — while the campaign records that no admitted oracle carries 2.5. SGLang-Omni implements nothing in this class. Neither correction closes #655:ltx_coreis what every LTX-2.5 correctness gate already runs against, and it is installed on no gate host, which is the real blocker under everyPENDINGspeed axis.Gates
scripts/agent-preflight.sh: exit 0. The earliercheck-env-doc/test_check_env_docred was check-env-doc REDs preflight on every branch: three VT_MOE_EXPERT_STREAM knobs landed undocumented in 3005447f8 #995, fixed by docs(ENG-EXPERT-STREAM): the three expert-streaming knobs are a deployment surface, not an allowlist entry (#995) #997, which this branch has merged.check-doc-checkpoint --commitper non-merge commit:d8466502f,aa74b0460,8ac1a6338,27875a6f8,0696bdff6all exit 0. Armed:--commit b5618b305exits 1 as required.issue-indexappend-only:git diff --numstat origin/main...HEADshows 12 rows added, 0 removed.Closes #1006. Files #1007, #1008, #1009, #1010, #1011, #1012, #1014, #1015, #1016, #1021, #1024.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]