LTX-2.5 device residency: the spike ranked the levers but nothing stages them into landable work

[localai-bot]

An LTX-2.5 render on GB10 barely uses the GPU. The investigation is done and is excellent;
what is missing is a row with implementation authority and a staged plan.

What already exists — this issue does NOT duplicate it

ltx25-decode-speed.md
is a SPIKE row that ranked the levers, read the oracles, and filed thirteen issues it
owns under ## Owed: #1007,
#1009,
#1010,
#1011,
#1012,
#1014,
#1015,
#1016,
#1021,
#1024,
#1040,
#1202,
#1210.

That row states plainly that it has no implementation authority and no fresh review.
It is a spike. Nothing in the tree stages its levers into landable, separately reviewed
work, and the levers span two memory-architecture changes on a box that OOM-reboots when
they go wrong.

The measured problem

• #1024 — with --device cuda the
  render stages 35.5 GiB onto the GPU and then does no compute on it. GPU utilization was
  exactly 0 in 321 of 347 samples, all 26 non-zero samples inside the staging window.
  From t=251 s onward, over 17 minutes, 0 in every sample while the process held exactly
  1.00 core of 20.
• #1087 — after
  #1041 landed threaded decode,
  57-66% of wall is one contiguous single-threaded phase measuring 1731 s and 1732 s
  across two rungs, i.e. essentially constant while voxel count grew 2.75x. Pure
  overhead on small renders; it does not amortise.
• #1007 — the video VAE decode has
  no device arm at all. Every oracle runs it GPU-resident.
• #1021 — DiT staging is 7.5 minutes
  at ~52 MiB/s with the GPU idle in 85% of samples: ~3,504 serial
  Alloc+copy+Synchronize round trips.
• #1015 — Ltx2WidenDitToF32 holds
  the bf16 originals AND the f32 copies at once, permanently: ~105.9 GiB of DiT weights on
  a 119 GiB box, ~37.9 GB of it dead.
• #1016 — the LTX-2.5 loaders never
  call MaybeReleaseSourcePages (0 hits against 15 other files under src/vllm), so the
  whole DiT file stays faulted resident beside its device copy. Measured at +10.83 GiB.

The thing that must come first, and why

#1040: none of the spike's
measurements has a retrievable evidence artifact. They are on dgx.casa, which stopped
answering; the spike marks rung 2's exit reason REMOTE_UNVERIFIED and says the same
applies to the passing numbers.

So the lever ranking this campaign would be built on rests on numbers nobody can
re-derive. Staging a multi-stage campaign on an unreproducible ranking is how a campaign
spends months on the wrong lever. Paired with
#1010 — a render emits ONE log line
in 2.5 hours, so no phase can be timed at all, and #1087 explicitly says "Do not guess it
from the duration" — the first stage is not a lever. It is the instrument.

Scope of this issue

Write and commit a staged campaign spec with the nine structured sections, taking
implementation ownership of the ranked levers. It must:

1. Not restate the spike. Cite it; do not duplicate its evidence, oracle analysis or
   provenance sections.
2. Stage the work so each stage lands and is reviewed separately, and is revertible
   without touching the next.
3. Put the instrument first: An LTX-2.5 render emits ONE log line in 2.5 hours, so no phase of it can be measured without re-deriving the timing from a memory sampler #1010 phase timestamps and LTX25-DECODE-SPEED's passing measurements have no retrievable evidence artifact, and the sampler cadence does not close #1040 retrievable artifacts,
   with an explicit re-derivation of the ranking afterward. A lever whose ranking does
   not survive re-measurement is not a lever.
4. State the hardware hazards as gates, not prose: a GB10 unified-memory OOM reboots the
   whole box, /tmp does not survive that reboot, and the fleet is leased through rc.
5. Name what would close each stage, including the outcomes where a lever is refuted.
   AGENTS.md forbids declaring a ceiling; a refuted lever names the next hypothesis.

Relation to #1164: that issue wants
the denoise loop captured in a CUDA graph. A graph collapses host launch dispatch, and
this render has almost none to collapse, so #1164 is blocked behind this campaign and its
decision point is a measurement — GPU-busy against wall once the loop is device-resident.
If it turns out GPU-bound, #1164 closes as a refutation the way
#1161 closed prefill capture.

No engine code in this issue. Spec only.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]

[localai-bot]

Correction to this issue's body, from the spec that closes it (PR #1271). Two claims I wrote here are wrong, and both were verified against the tree rather than argued.

1. "stages 35.5 GiB onto the GPU and then never uses it" does not identify its subject. The denoise is device-routed: src/vllm/multimodal/ltx2_video.cpp:847 selects Ltx2StreamDitToDevice, and :3864-3866 selects Ltx2DitForwardDevice, which calls CheckWeightsResident and drives 48 blocks of device ops. So #1024's headline reading — an unrouted denoise — is not what the code does, and the campaign's own headline measurement does not name the phase it measures. That is a stronger argument for instrument-first than the one this issue gave, not a weaker one.

2. #1015 does not gate device residency, contrary to the "Dependencies" framing above. Ltx2StreamDitToDevice refuses widen_to_f32 by name (src/vllm/model_executor/models/ltx2_loader.cpp:759-765), with the recorded reason that widening while staging "would move twice the bytes to save nothing". The bf16+f32 double-hold is therefore a host-arm defect. The issue that actually gates device headroom is #1016, because on unified memory the device copy and the dead file cache share the same 119 GiB.

The spec also adds a lever this issue missed entirely — the conditioning stage is host-only by type (Gemma4Weights → OwnedTensor → OwnedBytes, no device field), now filed as #1269 and staged W4.

The staged plan in ltx25-device-residency.md is authoritative over this issue body wherever the two disagree.