An LTX-2.5 render with --device cuda stages 35.5 GiB onto the GPU and then never uses it: GPU utilization is 0 in 321 of 347 samples, and every non-zero one is inside the staging window

[localai-bot]

Found by the rung-2 probe of ltx25-decode-speed.md §5 (#1006).

MEASURED on dgx.casa (kairos-17dd, GB10, driver 580.173.02, clocks.max.sm 3003 MHz, boot id 03717c9d-63c8-4652-a8fe-a63d012c5718, 20 cores), build 0e1bee42f arch 121a in vllmcpp-build:gb10, under $HOME/gpu.lock, at 320x192/25 frames with --device cuda and the full --encoder. Per-PID at 2 s, 347 samples over the first 1274 s:

• GPU utilization never exceeded 2%, and was exactly 0 in 321 of 347 samples.
• Every one of the 26 non-zero samples falls inside the DiT staging window (t <= 251 s), where it is incidental copy traffic rather than compute.
• From t = 251 s onward — over 17 minutes — GPU utilization is 0 in every sample, while the CUDA compute-app footprint sits flat at 36396 MiB (35.54 GiB) and the process holds exactly 1.00 core of 20 (utime +116.7 s over 117 s, then +142.2 s over 142 s, then +39.5 s over 39 s across consecutive windows).
• 0 frames written.

So the render stages 35.54 GiB of transformer weights onto a GPU it then does no compute on. --device cuda was passed, the staging clearly took the device path (the process's Anonymous stayed at 0.01 GiB through staging, so no f32 widening happened — i.e. im.on_device is true at src/vllm/multimodal/ltx2_video.cpp:786), and the weights are resident. Nothing uses them.

The disjunction this leaves, stated rather than guessed. Either the denoise loop is not taking Ltx2DitForwardDevice (ltx2_video.cpp:2946 vs the host Ltx2DitForward at :2948), or it is, and something before it occupies 17+ minutes of single-threaded host time. Both are defects, and the render emits nothing that timestamps a phase boundary (#1010), so this issue cannot currently distinguish them. That is the first thing to fix here.

Two instrument caveats, both material.

1. There is no positive control that utilization.gpu reports a high value for a real compute kernel on GB10. The counter is demonstrably live — it moved 0-3% during staging — but live is not correctly scaled, and this box already returns [N/A] for --query-gpu=memory.used. That control is owed and is one command under the lock: run any known GPU-saturating job and confirm the counter reads high. Until then a 0% reading is not on its own proof.
2. The CPU column does not depend on that counter and carries the claim independently. A full core of process time, a flat device counter, and no output are together only consistent with the host doing the arithmetic.

Relation to the other levers. #1007 (the video VAE decode has no device arm) is a known absence and is certainly part of this. This issue is broader: it says the GPU is idle for the whole post-load render, not only for the decode, which means #1007 alone may not close it. docs/USAGE.md:873-874 says "most of a 320x192/25f render is spent single-threaded in the host VAE decode at 0% GPU" — the 0% GPU is not a property of the decode phase; it is a property of the render after load.

Also relevant: the text encoder is host-only by declaration (ltx2_video.cpp:1684-1691 — a CPU queue is constructed at :1699 regardless of device, because ltx2_text_encoder.h is f32 by declaration), and it was measured multi-threaded at ~2.9 cores across 22 threads for ~150 s of this same run. So at least one large phase is host by design.

Listed under ## Owed in ltx25-decode-speed.md.

[localai-bot]

The positive control this issue owes has been taken, with a physical instrument rather than the utilization counter.

.agents/specs/ltx25-decode-speed.md §5 correctly flags utilization.gpu as uncalibrated on GB10 and lists a control as owed, because the 0%-in-321-of-347-samples finding rests on a counter that can read zero for structural reasons on this hardware. During the max-quality render run on 2026-08-16/17 (main 0b0b8900f), the same idle phase was measured by board power draw:

• 11.25-11.60 W at 50 °C, sm 0% — 15 samples on one rung and 10 more on a second, two independent runs.
• Taken while 36,396 MiB of DiT weights were resident on the device.
• Simultaneously on the host: main thread in state R, all 20 pool workers in state S, utime advancing at exactly 1.00 core, stime flat, and RSS byte-identical for minutes.

Power draw is not fooled the way a utilization counter is. An 11 W board with 35 GiB of weights resident, alongside a host burning exactly one core with every worker asleep, is a GPU doing nothing while a single host thread works — which is what this issue claims.

What it does not settle: it confirms the GPU is idle during that phase; it does not identify which phase, and it is still one-sided (our engine only, no oracle running the same workload). The phase itself is now the dominant cost of a render and is filed separately as #1087 — measured at 1731 s and 1732 s across two rungs whose voxel counts differ by 2.75x, i.e. constant in resolution.

So the ranking in the spec survives contact with the physical instrument, and the target is now sharper: not "the GPU is idle during the decode" but "the GPU is idle during a resolution-independent serial phase that #1041 did not touch and that now dominates".

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]