spec(LTX25-DECODE-SPEED): the decode has no device arm, and the 60 GiB was never in it

.agents/issue-index.md

@@ -269,4 +269,16 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#986](https://github.com/mudler/vllm.cpp/issues/986) | `LTX25-DFR-PIPELINE` | `DFRPipeline` (`dfr_pipeline.py` + `dfr_layout.py` @ `fd4ded7f`) has no representation in this tree — `git grep -i dfr` over `src include tests examples docs` returns ZERO product hits, against 87 `ltx2` hits in `ltx2.cpp` alone as the positive control ([#604](https://github.com/mudler/vllm.cpp/issues/604)). It matters beyond its own feature because it is the **ONLY** upstream consumer of the temporal x2 latent upsampler this project already ships: [`ltx25-temporal-upsampler.md`](specs/ltx25-temporal-upsampler.md) section 7 records the operator as ported, loader-parsed and gated but *"not reachable from any shipped pipeline"*, and `docs/FEATURES.md` carries that as `Temporal x2 ups gated, UNDRIVEN`. No issue tracked that state — `undriven` returned zero hits across open and closed issues. DFR also needs the generated-keyframe-slot READBACK that [#920](https://github.com/mudler/vllm.cpp/issues/920) refused by name and left owed after it CLOSED, so the debt had a spec bullet and no open issue; it needs the LAYOUT and the EXTRACTION but **not** the standalone single-frame decode, because DFR never decodes its slots — it hands them to the spatial upsampler (`dfr_pipeline.py:348`) and feeds them back as `initial_keyframes` (`:364`). Spec [`ltx25-dfr-pipeline.md`](specs/ltx25-dfr-pipeline.md). Campaign [#644](https://github.com/mudler/vllm.cpp/issues/644) | feature |
 | [#995](https://github.com/mudler/vllm.cpp/issues/995) | `ENG-EXPERT-STREAM` | `check-env-doc` and `test_check_env_doc` RED in `scripts/agent-preflight.sh` on **every** branch based on `3005447f8`: `VT_MOE_EXPERT_STREAM`, `VT_MOE_EXPERT_STREAM_SLOTS` and `VT_MOE_EXPERT_STREAM_SLOT_BYTES` are read from `src/vllm/model_executor/models/qwen3_5.cpp` (`:5145`, `:5164` @ `3005447f8`) and appear in neither `docs/ENVIRONMENT.md` (`grep -c` returns **0**) nor `scripts/env-doc-allowlist.txt`. Arrived with [#993](https://github.com/mudler/vllm.cpp/pull/993). Found while gating [#986](https://github.com/mudler/vllm.cpp/issues/986) and proved pre-existing with a matched-arm check rather than asserted: the three sites are in a file that branch does not touch, and `git diff origin/main...HEAD | grep '^+.*VT_MOE_EXPERT'` returns nothing. NOT fixed in flow: documenting a knob means stating its default and when to touch it, and the expert-streamer's slot accounting belongs to the row that added it, so a plausible-sounding entry written by a passer-by is how `docs/ENVIRONMENT.md` stops being trustworthy. It is a PRE-FLIGHT gate, so it fails before every edit and presents to each author as a red their own diff caused, which is the shape [#965](https://github.com/mudler/vllm.cpp/issues/965) and [#968](https://github.com/mudler/vllm.cpp/issues/968) both took | bug |
 | [#987](https://github.com/mudler/vllm.cpp/issues/987) | `LTX25-RETAKE` | Two `ltx-2.5` refusal messages state reasons that are no longer true. (a) `src/vllm/multimodal/ltx2_video.cpp:1608 @ 0e1bee42f` says "nothing reads `ref_video_dir` at all", and MiniMax-H3 has always consumed the directory in full (`ReadReferenceClipChw`, `src/vllm/multimodal/minimax_h3_video.cpp:135 @ 0e1bee42f`, called at `:650`); [#975](https://github.com/mudler/vllm.cpp/issues/975) inherited the wider claim from this message. The claim that holds is narrower: the LTX-2.5 engine never reads the directory's CONTENTS. (b) `ltx2_video.cpp:1636-1638 @ 0e1bee42f` says "there is no AUDIO_VAE_ENCODER key filter", and `c2019b0e3` landed `Ltx2AudioVaeEncoderKeyRules()` (`include/vllm/model_executor/models/ltx2_audio_input.h:73 @ 0e1bee42f`) with a live call through `Ltx2EncodeAudioToLatent`. Both rewritten in the `WHAT IS *NOT* THE REASON` shape in the same flow, with one assertion tied to the LOCAL fact that the LTX side now reads the directory | bug |
+| [#1006](https://github.com/mudler/vllm.cpp/issues/1006) | `LTX25-DECODE-SPEED` | LTX-2.5 render speed has never been attributed on any axis: `docs/BENCHMARKS.md` carries one LTX line, under `## Open gaps`. The shipped video VAE decode is the **CPU reference arm** and production executes it — `src/vllm/multimodal/ltx2_video.cpp:3258 @ 332aed738` calls `Ltx2VideoDecodeStreaming`, reaching `Ltx2ConvVideoDecode` via `ltx2_video_vae_tiled.cpp:113,369 @ 332aed738` — while the file itself says `src/vllm/model_executor/models/ltx2_video_vae.cpp:46-49 @ 332aed738` "no memory or throughput number should be taken from it". One 448x256/25f decode is ~7.25 TFLOP over 42 convs (COMPUTED from the LTX-2.5 conv VAE config in the checkpoint header) and measures 2681.02 s — ~2.7 GFLOP/s. Owning row for the ranked levers in [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) | feature |
+| [#1007](https://github.com/mudler/vllm.cpp/issues/1007) | — | The LTX-2.5 video VAE decode has **no device arm at all**: `vt::OpId::kLtx2` (`include/vllm/model_executor/models/ltx2_kernels.h @ 332aed738`, `src/vt/cuda/cuda_ltx2.cu @ 332aed738`) is the DiT device-forward glue — seven ops, no convolution — and nothing the decode reaches. Not an unwired path; the arm does not exist. Every oracle runs this decode GPU-resident and decides placement at build time: Lightricks `packages/ltx-pipelines/src/ltx_pipelines/utils/blocks.py:1139` + `packages/ltx-core/src/ltx_core/loader/single_gpu_model_builder.py:273 @ fd4ded7f2`, SGLang `.../ltx_2/decoding_av.py:71 @ f63458b5b`, vLLM-Omni `vllm_omni/diffusion/models/interface.py:92 @ a4ea67a21` ("VAE(s) (always on GPU)"), diffusers `ltx2_diffusion_decoder.py:208-209 @ 3a2f35d4e` ("No CPU path"). Lever 1, ranked first on magnitude and last on cost. Listed under `## Owed` in [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) | feature |
+| [#1008](https://github.com/mudler/vllm.cpp/issues/1008) | — | The LTX-2.5 video VAE decode accumulates in **f64** at 8 sites (`src/vllm/model_executor/models/ltx2_video_vae.cpp:165,201,303,312,546,570,579,916 @ 332aed738`, with 29 `static_cast<double>`) and stores **f32 NCDHW** (`Volume::At`, `:73-75 @ 332aed738`). No oracle accumulates in f64 anywhere on this path — Lightricks `model/video_vae/conv_video_decoder.py:282-284` and `model/common/normalization.py:32-40 @ fd4ded7f2`, diffusers `autoencoder_kl_ltx2.py:50-59 @ 3a2f35d4e`, SGLang `configs/pipeline_configs/ltx_2.py:189 @ f63458b5b` (bf16, deliberately overriding an fp32 base default) — and upstream's default-on fast path is `channels_last_3d` for weights AND activations (`memory_efficient_decode.py:617-627,655-656 @ fd4ded7f2`). No existing gate can see it: the golden generator casts every upstream parameter to f32, so the oracle itself runs f32 (`ltx2_video_vae.cpp:41-44 @ 332aed738`). Lever 2, the cheapest large win. Listed under `## Owed` in [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) | feature |
+| [#1009](https://github.com/mudler/vllm.cpp/issues/1009) | — | The LTX-2.5 video VAE decode is **single-threaded on a 20-core box**. `ParallelForRows` (`src/vt/cpu/cpu_threadpool.cpp:413 @ 332aed738`) is synchronous and used by 10+ CPU kernels (`cpu_conv2d.cpp:78`, `cpu_layernorm.cpp:53`, `cpu_paged_attn.cpp:185`, `cpu_quant_gemm.cpp:191`, `cpu_ops.cpp:28 @ 332aed738`); zero are in the decode, whose loop nest at `ltx2_video_vae.cpp:161-164 @ 332aed738` is perfectly nested and parallel over `oc`. MEASURED from pre-existing evidence: the completed 320x192/49f render of 2026-08-15 (`~/work/ltx25-e2e/render8-console.log` on `dgx.casa`) holds a 1-minute load average of 1.0-1.3 for ~2h07m of its 2h23m wall — about 89%. A local seam, not an upstream mirror; no oracle has a host decode. Lever 3. Listed under `## Owed` in [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) | feature |
+| [#1010](https://github.com/mudler/vllm.cpp/issues/1010) | — | An LTX-2.5 render emits **one** engine log line in 2.5 hours: `~/work/ltx25-e2e/ladder-075434/512x320_121f/run.log` on `dgx.casa` is 16 lines for a 2h29m run, 13 of them the CUDA container banner. Attributing 89% of a render's wall to a single-threaded phase (#1009) therefore required reading an external memory sampler's load-average column and correlating it with file mtimes. Per-phase wall and peak memory are owed from the render path; the streaming chunk callback (`src/vllm/multimodal/ltx2_video.cpp:3262-3283 @ 332aed738`) already carries `chunk.first_frame` and is the natural site. No speedup — it is the precondition for measuring #1007, #1008 and #1009, and should land first. Lever 6. Listed under `## Owed` in [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) | feature |
+| [#1011](https://github.com/mudler/vllm.cpp/issues/1011) | — | `memory_efficient_decode.py` is **ON BY DEFAULT** upstream (`memory_efficient: bool = True`, `packages/ltx-pipelines/src/ltx_pipelines/utils/blocks.py:1059 @ fd4ded7f2`, sole caller `:1090-1095`) and unported here — but it is **NOT** the unattributed ~59 GiB, and [`ltx25-resolution-envelope.md`](specs/ltx25-resolution-envelope.md):353 calling it "the obvious" candidate is refuted: summing every intermediate the conv decoder ever produces at 448x256/25f in f32 with **no frees at all** gives 9.649 GiB (COMPUTED), 6x short. Still owed for byte traffic and memory format — workspace buffers (`memory_efficient_decode.py:108-114 @ fd4ded7f2`) replacing a per-conv `repeat`+`concatenate` (`model/video_vae/convolution.py:306-311`), in-place chunked Conv3d (`:122-204`), free-before-conv (`:234-248`), and NDHWC (`:617-627`). The shipped LTX-2.5 conv VAE is non-causal, so all four apply. Lever 4. Listed under `## Owed` in [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) | feature |
+| [#1012](https://github.com/mudler/vllm.cpp/issues/1012) | — | Record defect: `diffusers` @ `3a2f35d4e` **implements LTX-2.5**, both decode arms — `AutoencoderKLLTX2Video` (`src/diffusers/models/autoencoders/autoencoder_kl_ltx2.py:1025`) and `LTX2VideoDiffusionDecoderModel` (`ltx2_diffusion_decoder.py:700`, docstring `:702` "introduced in LTX-2.5", pipeline `pipeline_ltx2_diffusion_decode.py:27`) — and it is already in the AGENTS.md oracle table and already pinned in [`oracles/diffusers.md`](oracles/diffusers.md). `roadmap_v1.md:92 @ 332aed738` and `docs/BENCHMARKS.md:465 @ 332aed738` both record that no admitted oracle carries 2.5. The generalization came from vLLM-Omni pinning `diffusers==0.38.0` and importing only `diffusers.pipelines.ltx2`. Does NOT close [#655](https://github.com/mudler/vllm.cpp/issues/655): `ltx_core` is what every LTX-2.5 CORRECTNESS gate already runs against. Neither is installed on `dgx.casa`. Also corrects SGLang-Omni, which implements nothing in this class. Listed under `## Owed` in [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) | bug |
+| [#1014](https://github.com/mudler/vllm.cpp/issues/1014) | — | The ~59 GiB an LTX-2.5 render loses is **not in the decode**, established twice: MEASURED heap peak 361.72 MiB by exact `operator new` accounting ([`ltx25-tiled-decode.md`](specs/ltx25-tiled-decode.md) `## Outcome` item 2), and a COMPUTED 9.649 GiB ceiling assuming nothing is ever freed. Hypothesis, NOT closed: model residency — `docs/USAGE.md:862-864 @ 332aed738` already accounts for ~68 GiB staged and resident before any decode instruction, and the 320x192/49f render trace shows ~72 GiB acquired in the first ten minutes and held flat for two hours. A 2 s one-clock trace of `MemAvailable` + `VmRSS`/`Anonymous` + the CUDA compute-app footprint across the load/denoise/decode boundary settles it; if the fall is at the decode boundary instead, the next hypothesis is a CUDA or `mmap` mapping. Also owes the `docs/USAGE.md:873-874 @ 332aed738` "inside the decode" correction. Lever 5. Listed under `## Owed` in [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) | bug |
+| [#1015](https://github.com/mudler/vllm.cpp/issues/1015) | — | `Ltx2WidenDitToF32` (`src/vllm/model_executor/models/ltx2_loader.cpp:694-710 @ 332aed738`) holds the **bf16 originals AND the f32 copies at once, permanently**: `:707` does `checkpoint.storage.push_back(std::move(widened))` — an APPEND — and repoints `view.data`, but nothing drops the original `storage` entry. At 18.95B params that is 2+4 bytes each = **~105.9 GiB COMPUTED on a 119 GiB box**, ~37.9 GB of it dead the instant the loop finishes. Reached on every host-arm load via `src/vllm/multimodal/ltx2_video.cpp:786 @ 332aed738` (`widen_to_f32 = !on_device`). `ltx2_video.h:109 @ 332aed738` quotes "~76 GB" — the f32 half alone. Found while attributing [#1014](https://github.com/mudler/vllm.cpp/issues/1014); it is NOT that fall, which follows a flat plateau this cost precedes. Fix must not free a buffer another view still points at (same class as [#949](https://github.com/mudler/vllm.cpp/issues/949)). Listed under `## Owed` in [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) | bug |
+| [#1016](https://github.com/mudler/vllm.cpp/issues/1016) | — | The LTX-2.5 loaders **never call `MaybeReleaseSourcePages`**: 0 hits in `src/vllm/model_executor/models/ltx2_loader.cpp` and `src/vllm/multimodal/ltx2_video.cpp @ 332aed738`. Positive control: the symbol appears in **15** other files under `src/vllm @ 332aed738` (`muse_glimmer_weights.cpp`, `phi_weights.cpp`, `gemma4_weights.cpp`, `nemotron_h_weights.cpp`, `kimi_linear_weights.cpp`, `qwen3_5_dense_weights.cpp`, `olmo2_weights.cpp` among them), so the symbol and path set are right and the calls genuinely are not there. It belongs in the device staging loop `ltx2_loader.cpp:738-756 @ 332aed738`, which walks one tensor at a time and `Synchronize`s at `:749`, so each source range is provably dead per iteration. On GB10 the staged copy and the file pages share one 119 GiB pool. Unlike [#1015](https://github.com/mudler/vllm.cpp/issues/1015) this is NOT excluded as a contributor to [#1014](https://github.com/mudler/vllm.cpp/issues/1014), because `MemAvailable` discounts reclaimable pages and the `Anonymous`-vs-`Rss_File` split has never been measured here. Listed under `## Owed` in [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) | bug |
+| [#1021](https://github.com/mudler/vllm.cpp/issues/1021) | — | LTX-2.5 **DiT device staging takes 450 s — 7.5 minutes — at ~52 MiB/s**, and no record names the phase. MEASURED on `dgx.casa` (GB10, boot id `03717c9d-63c8-4652-a8fe-a63d012c5718`, build `0e1bee42f`, under `$HOME/gpu.lock`), per-PID at 2 s over 192 samples: CUDA compute-app footprint 4.22 -> **35.20 GiB**, GPU utilization mean **0.2%** with zero in **164/192** samples, CPU **0.15 cores of 20**, `Anonymous` +0.01 GiB. Neither GPU-bound nor CPU-bound. The shape is `src/vllm/model_executor/models/ltx2_loader.cpp:738-756 @ 332aed738`: ~3,504 tensors, each a raw `cudaMalloc` (`backend.Alloc` at `:747`, `src/vt/cuda/cuda_backend.cu:77-81 @ 332aed738`) followed by a full `backend.Synchronize(queue)` at `:749`, serialized against the host read. The plateau at 36396 MiB = 35.54 GiB lands within 1% of the 35.32 GiB the loader contract predicts. Same loop as [#1016](https://github.com/mudler/vllm.cpp/issues/1016); a row should take both. Listed under `## Owed` in [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) | feature |
+| [#1024](https://github.com/mudler/vllm.cpp/issues/1024) | — | An LTX-2.5 render invoked with `--device cuda` **stages 35.54 GiB onto the GPU and then never uses it**. MEASURED on `dgx.casa` (GB10, boot id `03717c9d-63c8-4652-a8fe-a63d012c5718`, build `0e1bee42f`), 320x192/25f, per-PID at 2 s over 347 samples: GPU utilization never exceeded 2%, was **exactly 0 in 321/347**, and **every one of the 26 non-zero samples falls inside the DiT staging window** (t<=251 s). From t=251 s for 17+ minutes it is 0 in every sample while the compute-app footprint sits flat at 36396 MiB and the process holds **exactly 1.00 core of 20**; 0 frames written. Staging took the device path (`Anonymous` stayed at 0.01 GiB, so no f32 widening — `src/vllm/multimodal/ltx2_video.cpp:786 @ 332aed738`). Leaves a disjunction this issue cannot yet split, because nothing timestamps a phase boundary ([#1010](https://github.com/mudler/vllm.cpp/issues/1010)): either the denoise is not taking `Ltx2DitForwardDevice` (`ltx2_video.cpp:2946` vs `:2948 @ 332aed738`), or it is and something before it burns 17+ minutes single-threaded. Broader than [#1007](https://github.com/mudler/vllm.cpp/issues/1007): the 0% GPU is a property of the whole post-load render, not of the decode phase, so `docs/USAGE.md:873-874 @ 332aed738` understates it. **Owed: a positive control that `utilization.gpu` reads high for a real kernel on GB10** — this box already returns `[N/A]` for `--query-gpu=memory.used`. Listed under `## Owed` in [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) | bug |
 | [#995](https://github.com/mudler/vllm.cpp/issues/995) | `ENG-EXPERT-STREAM` | `check-env-doc` and `test_check_env_doc` are RED on `origin/main`, so every branch cut from it inherits a preflight failure its own diff did not cause: `VT_MOE_EXPERT_STREAM`, `VT_MOE_EXPERT_STREAM_SLOTS` and `VT_MOE_EXPERT_STREAM_SLOT_BYTES` are read from `src/vllm/model_executor/models/qwen3_5.cpp` (`:5145`, `:5195`, `:5189` @ `4496ef196`) and appear in neither `docs/ENVIRONMENT.md` nor `scripts/env-doc-allowlist.txt`. They arrived with the `ENG-EXPERT-STREAM` W4 wiring commit `3005447f8` ([#993](https://github.com/mudler/vllm.cpp/pull/993)). Same shape as [#965](https://github.com/mudler/vllm.cpp/issues/965)/[#968](https://github.com/mudler/vllm.cpp/issues/968): a pre-existing red presenting to each author in turn as their own. Fixed by DOCUMENTING all three in `docs/ENVIRONMENT.md`, not by allowlisting them — a knob that changes the host memory a MoE decode reserves and disables the default-on grouped-MoE path is a deployment surface, not a kernel-internal micro-tuning switch | bug