record(LTX25-RESOLUTION-ENVELOPE): 704x448/25f completes, and the size the page said could not (#1088)

.agents/issue-index.md

@@ -305,3 +305,4 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#960](https://github.com/mudler/vllm.cpp/issues/960) | `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm` | `vt::QuantFp8Static`'s ONLY CUDA registration lived at `src/vt/cuda/cuda_matmul_fp8_cutlass.cu:376` (@ `0e1bee42f`), and `CMakeLists.txt:1668` compiles that translation unit only when `VT_CUTLASS_FP8_ARCHS` is non-empty — yet the kernel body has ZERO cutlass tokens (`:353-370`): it is `out[i] = e4m3(x[i] * (1/input_scale))`, a grid-stride elementwise convert. So on every CUDA arch outside the cutlass-fp8 cell — sm_110/Thor is the measured one, and `cutlass-fp8: DISABLED for [110]` is that arch's DOCUMENTED NORMAL PROFILE, not a misconfiguration — `OpId::kQuantFp8Static` was not registered for `DeviceType::kCUDA` at all. Nothing refused first: the GEMM partner `kMatmulFp8CublasLt` IS registered unconditionally (`src/vt/cuda/cuda_matmul.cu:920`), so `MatmulFp8CutlassD`'s guard passed, and the missing quant then resolved through `src/vt/op_provider.cpp:501` to the portable CPU reference tier — eligible because `CudaBackend::UnifiedMemory()` is true — which dereferenced DEVICE pointers on the host and SIGSEGV'd one call later under a banner reading "correct but slow". Fixed by relocating the registration to a new unconditionally-compiled TU `src/vt/cuda/cuda_quant_fp8.cu`, which restores upstream's own partition (vLLM builds `static_scaled_fp8_quant` from the unconditional `VLLM_EXT_SRC` list and gates only its cutlass `scaled_mm` sources). This removes one live INSTANCE of [#844](https://github.com/mudler/vllm.cpp/issues/844) and does not address its class, which stays open. Unblocks the FP8 W8A8 arm on every non-cutlass CUDA arch — the base [#810](https://github.com/mudler/vllm.cpp/issues/810)/[#517](https://github.com/mudler/vllm.cpp/issues/517) A2-Q1 needs, where 46 FP8 mamba projections are 36.6% of decode bytes. Spec [`vt-fp8-quant-arch-gate.md`](specs/vt-fp8-quant-arch-gate.md) | bug |
 | [#989](https://github.com/mudler/vllm.cpp/issues/989) | `VT-FP8-QUANT-ARCH-GATE` | `scripts/check-pr-size.py`'s `classify_path` has no entry for `.agents/reachability.md` (added by `POLICY-NOTHING-LANDS-DEAD`, [#888](https://github.com/mudler/vllm.cpp/issues/888) @ `8f49ac3be`), and it FAILS CLOSED, so `pr-size` — a REQUIRED check — refuses every pull request that touches that guide, and `tests/scripts/test_check_pr_size.py` has been red on `main` ever since. Red SILENTLY: that suite is wired into no CI job and is not in `agent-preflight.sh`'s `SUITES`, so the only thing that ever loads it is `check-pr-size`'s own executable-evidence contract, which fires only when a PR edits a checker — the red is reachable exclusively by the next person who must touch that file, and presents to them as their own breakage (the [#584](https://github.com/mudler/vllm.cpp/issues/584)/[#965](https://github.com/mudler/vllm.cpp/issues/965) shape). Third instance of the class after [#856](https://github.com/mudler/vllm.cpp/issues/856) (`issue-index.md` + the style guides) and [#668](https://github.com/mudler/vllm.cpp/issues/668) (`.agents/oracles/*`), both fixed in flow by the row that tripped over them. FIXED IN FLOW while landing [#960](https://github.com/mudler/vllm.cpp/issues/960), which could not register its new checker's creation mutation without touching `check-pr-size.py` at all. NOT fixed: wiring that suite into CI, which is its own change and would red `main` until this landed | bug |
 | [#1066](https://github.com/mudler/vllm.cpp/issues/1066) | `ENG-EXPERT-STREAM` | `Qwen35ExpertStream` (`src/vllm/model_executor/models/qwen3_5.cpp`) is a **process-lifetime singleton** and keyed its slot cache on `(TowerId(base), expert)`, where `base` is the expert tower's host buffer **ADDRESS**. Its own comment stated the premise and drew the wrong conclusion: "A tower's identity is its base pointer, which is stable for the model's life". The premise is true; the conclusion does not follow, because the CACHE is not scoped to one model's life. Free a model, load another, and the allocator hands the new towers addresses the old ones held, so the new model's expert resolves to an entry filled from a DIFFERENT checkpoint — returned as a HIT, which by contract moves no bytes, so no counter moves and nothing downstream has anything to observe. MEASURED on two synthetic 4-layer/4-expert MoE models in one process, instrumenting `KqExpertSlice` to `memcmp` each returned slot against the tower slice it claims to be: **24 towers occupied 21 distinct addresses, and 20 of 222 slices returned another tower's bytes**; end to end the two arms disagreed on all 160 logits while each arm was internally deterministic (0 differing values on a repeat), which rules out nondeterminism. Invisible to every existing test of this row by construction, because all of them build the cache, store and streamer by hand and none runs two models through the production seam. Reachable by any process that loads a model, releases it, and loads another. Fixed by `OwnedTensor::TowerUid()`, a lazily assigned process-unique counter stamped on the tensor and re-stamped when `bytes` moves (so a copy cannot inherit an identity along with a different buffer); a counter cannot collide because it never goes backwards. Found and fixed while repairing the F1-F11 wiring review for [#912](https://github.com/mudler/vllm.cpp/issues/912). Spec [`expert-streaming.md`](specs/expert-streaming.md) | bug |
+| [#1088](https://github.com/mudler/vllm.cpp/issues/1088) | `LTX25-RESOLUTION-ENVELOPE` | `docs/USAGE.md` published "Measured to complete on one GB10 = 320x192 at 25 frames" beside "Measured NOT to complete = 448x256 at 25 frames, the denoise finishes, then the decode loses about 59 GB in 24 s", and `docs/BENCHMARKS.md` said the same. Two renders on `dgx.casa` on 16 to 17 August 2026 against `main` `0b0b8900f`, which carries [#1041](https://github.com/mudler/vllm.cpp/issues/1041) threaded decode, [#1032](https://github.com/mudler/vllm.cpp/issues/1032) T2A and [#1036](https://github.com/mudler/vllm.cpp/issues/1036) f32 decode accumulators, in container `vllmcpp-build:gb10`, `Release`, `VLLM_CPP_CUDA=ON`, arch `121a`, `TRITON=ON`, CUTLASS absent so FlashAttention-2 was not built, `VLLM_CPP_CPU_THREADS=20`, NVFP4 transformer, no `--allow-unported`, **COMPLETED 448x256/25f in 3085 s and 704x448/25f in 4231 s**. 1024x576/25f was not attempted to completion because another session claimed the box. That is scheduling and not an envelope, so 704x448 is NOT a ceiling. The ~59 GiB cliff did NOT recur under a 2 s memory guard that would have seen it. `MemAvailable` floors were 38.96 GiB over 1289 samples at 448x256 and 38.89 GiB over 1743 samples at 704x448, with ZERO samples under 34 GiB on either, peak use of 80 of 119 GiB, and no reboot. The 704x448 artifact was verified rather than inferred from an exit code: 25/25 distinct frame md5s, 0 near-uniform and 0 near-black frames, adjacent-frame mean absolute difference 4.381 against a uniform-noise reference of 85.3 on the same shape, 0/24 zero-motion pairs, and audio at 48 kHz stereo, 1.010 s, -37.29 dBFS, 20/20 windows above threshold. NOT claimed: ONE run per geometry on a contended shared box with no oracle on either side, so two points establish no scaling law. The 59 GB observation is KEPT and attributed to its own run, which is rung F1 in `benchmark-record.md`, a prompt-embeds render with no text tower that an ARMED WATCHDOG ended at 13.77 GiB against an 18 GiB floor rather than the engine failing. Attributing it stays open as [#1014](https://github.com/mudler/vllm.cpp/issues/1014). The claim that most of a 320x192/25f render is spent in the host VAE decode is superseded too: after #1041 a resolution-independent ~1731 s single-threaded phase, measured at 1731 s and 1732 s across two rungs whose voxel counts differ 2.75x, is 57 to 66% of wall, filed as [#1087](https://github.com/mudler/vllm.cpp/issues/1087). FIXED IN FLOW in `docs/USAGE.md`, `docs/BENCHMARKS.md` and [`ltx25-resolution-envelope.md`](specs/ltx25-resolution-envelope.md) section 4.1 | bug |

.agents/specs/ltx25-resolution-envelope.md

@@ -360,6 +360,66 @@ Set against upstream's own defaults — 1024x1536 and 1088x1920 at 121 frames 
 320x192 practical ceiling is the story, and the envelope states both numbers next
 to each other rather than only the legal one.
 
+### 4.1 Superseded on 16 to 17 August 2026: 448x256 completes, and so does 704x448
+
+[#1088](https://github.com/mudler/vllm.cpp/issues/1088). The bullets above are
+**kept as written** because they were true of the runs that produced them; this
+subsection records what replaced them, and `docs/USAGE.md` now publishes the
+newer envelope. Deleting the old bullets would remove the evidence the newer
+result is measured against.
+
+Measured on `dgx.casa` against `main` `0b0b8900f`, which carries
+[#1041](https://github.com/mudler/vllm.cpp/issues/1041) (threaded decode),
+[#1032](https://github.com/mudler/vllm.cpp/issues/1032) (T2A) and
+[#1036](https://github.com/mudler/vllm.cpp/issues/1036) (f32 decode
+accumulators). Container `vllmcpp-build:gb10`, `Release`, `VLLM_CPP_CUDA=ON`,
+arch `121a`, `TRITON=ON`, CUTLASS absent so FlashAttention-2 was not built, which
+is like for like with the earlier renders. `VLLM_CPP_CPU_THREADS=20`. NVFP4
+transformer. No `--allow-unported`.
+
+| Geometry | Result | Wall |
+|---|---|---|
+| 448x256 / 25 frames | **completed** | 3085 s |
+| **704x448 / 25 frames** | **completed** | 4231 s |
+| 1024x576 / 25 frames | not attempted to completion, another session claimed the box | n/a |
+
+The 1024x576 rung stopped for scheduling and not for memory or an envelope, so
+**704x448 is not a ceiling**, on the standard AGENTS.md applies to every
+measured limit and that §4 applied to 320x192.
+
+**The ~59 GiB cliff did not recur, under an instrument that would have seen it.**
+A memory guard at a 2 s cadence: the 448x256 rung floors `MemAvailable` at
+**38.96 GiB** over 1289 samples, the 704x448 rung at **38.89 GiB** over 1743
+samples, and **zero** samples on either fall below 34 GiB. Peak use was 80 of
+119 GiB, and the box did not reboot. This does not close
+[#1014](https://github.com/mudler/vllm.cpp/issues/1014), which owns attributing
+the original fall; it records that the fall is not reproducible on this build.
+Note also what the original run actually was: `benchmark-record.md` rung F1 is a
+prompt-embeds render with no text tower that ended in a **watchdog kill** at
+`avail_kB=13774472` against an armed 18 GiB floor, not in an engine failure.
+
+**The 704x448 artifact was verified, not assumed from an exit code.** Global mean
+90.34, std 60.54, per-frame variance 3630-3706, **0 near-uniform and 0 near-black
+frames**; **25/25 distinct md5s**, adjacent-frame mean-abs-diff 4.381 against a
+uniform-noise reference of 85.3 on the same shape, **0/24 zero-motion pairs**;
+audio 48 kHz stereo, 1.010 s, RMS **-37.29 dBFS**, 20/20 windows above threshold.
+The mp4 is at `benchmarks/media/ltx25-704x448-25f-audio.mp4` on the render host
+and is gitignored by `.gitignore:35` (`*.mp4`), so it is not committed.
+
+**What is not claimed.** One run per geometry, on a shared box that was
+contended, with no oracle on either side. Two points do not establish a scaling
+law. Nothing here says what the frames depict. That question is still the one
+§0 and `docs/USAGE.md` leave open.
+
+**The bound moved off the decode.** §4 named the decode's single-threaded
+throughput as owed measurement. #1041 answered it, and the same run shows the
+position was inherited rather than removed: a **resolution-independent ~1731 s
+single-threaded phase** (1731 s and 1732 s across two rungs whose voxel counts
+differ 2.75x) is now 57-66% of wall.
+[#1087](https://github.com/mudler/vllm.cpp/issues/1087) owns identifying it, and
+the sampler classified by CPU-time rate rather than by symbol, so what is
+measured is a duration and a scaling law and **not** a named function.
+
 ## 5. Tests
 
 Red-first, entering through the production entry point per
@@ -459,7 +519,13 @@ takes a different path entirely.
   recipe that ships. Covered by #644 row scope; not separately filed, because
   #644 already owns "close every refused arm".
 - Attribution of the 60 GiB decode loss and the single-threaded decode
-  throughput (§4). Both need the GPU; both are measurement rows.
+  throughput (§4). Both need the GPU; both are measurement rows. **Both moved,
+  and neither closed. See §4.1.** The decode throughput was answered by
+  [#1041](https://github.com/mudler/vllm.cpp/issues/1041), which handed the
+  position to an unidentified serial phase
+  ([#1087](https://github.com/mudler/vllm.cpp/issues/1087)). The 60 GiB loss did
+  not reproduce on `0b0b8900f` under a 2 s guard, which is not an attribution:
+  [#1014](https://github.com/mudler/vllm.cpp/issues/1014) still owns it.
 - **The lcm form of the divisor** (§3.1). `max_spatial_downscale()` is the
   maximum where the correct quantity is the least common multiple of the phase
   downscales. The two agree on every shipped recipe and part on a recipe with

docs/BENCHMARKS.md

@@ -484,7 +484,7 @@ built on it rather than keeping the flattering one.
 | Kimi-Linear-48B-A3B (KDA+MLA+MoE) | **RUNNER FOLD LANDS (ROW 7, §21, #122): engine==CLI 128/128 byte-identical; vs golden 122/128 (near-tie profile); FA2 MLA default-ON; SACRED green.** Server 19.0 tok/s wall; CLI 18.93 reproduced | vLLM ~21 (#111 floor; in-session re-measure ABORTED by GB10 reboot at util 0.82, §21): **~0.90×**, >= vLLM NOT met; residual = KDA host islands + grouped MoE + decode graph |
 | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin |
 | MiniMax-H3 FP4 speed (W-FP4a) | **Measured GB10 (`row/H3-FP4-GPU-E2E`).** Marlin W4A16 byte-exact vs bf16; fp4 a memory win, 0.8x bf16/forward. Real-ckpt fp4-resident e2e RUNS (mp4/wav) | fp4 speed CLOSED. bf16-vs-quant A/B: ENCODER half MEASURED (§8.15), DiT half NOT (no bf16 render exists). Detail: benchmark-record + spec §8 |
-| LTX-2.5 axes | Speed `PENDING` (vllm-omni#6066 has no native 2.5), binding oracle too. **SIZE: 320x192/25f completes on GB10, 448x256 does not**; that render was REGISTER-conditioned, not prompted | Wall is the HOST VAE decode, not the pool: drain returns 0.11 GiB, byte-inert. 2 baselines UNRESOLVED (lock). A real-checkpoint PROMPTED render is OWED. Decode THREADED ~9x at 16-20 workers, contended box (#1009) |
+| LTX-2.5 axes | Speed `PENDING` (vllm-omni#6066 has no native 2.5), binding oracle too. **SIZE: 704x448/25f and 448x256/25f both COMPLETE on GB10 (4231 s, 3085 s)**; one run each, contended box, no oracle, no ceiling (#1088) | Wall is NOT the VAE decode after #1041/#1009: a ~1731 s serial phase FLAT in resolution is 57-66% (#1087). ~59 GiB cliff did NOT recur (floor 38.9 GiB). 2 baselines UNRESOLVED (lock). PROMPTED real-ckpt render OWED |
 | MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`) | **Every axis vs the reference stays `PENDING`.** A PARTIAL device arm now exists (#672): only the 8.6B LM runs on the accelerator, so this is an internal two-arm number and NOT a parity ratio | Denominator: SGLang-Omni `748a0b43` in its production configuration (both CUDA graphs, compiled DIT and DAV, batched seeded sampling) |
 | MiniMax-Music3 device arm, Jetson Thor sm_110 (#672) | `--device 1` vs `--device 0`, same request/seed, idle box: 2 AR frames **846.6 vs 835.1 s (1.014x SLOWER)**; 10 frames **1430.4 vs 1512.1 s (0.946x)**. Fit: **-11.65 s/frame, +34.8 s fixed** | A third duration (the fit has no residual), and moving the depth decoder + DiT + vocoder, which are 5 of 6 stages and still host scalar loops |
 | MiniMax-Music3 CPU host kernels, x86-64 20-core (#672) | KERNEL A/B at the vocoder's real geometry, min of 5 interleaved rounds: convolution chain **13.36 -> 1.25 s, 10.7x**; `Conv1d` 12.03x, `LinearNoBias` 10.88x. Output fingerprints IDENTICAL on both arms | e2e pair VOID (cold CIFS cache; a foreign `ctest` at load 76.6) and re-running. Stages 0/1 only ~2x: the pivot trades WEIGHT locality for accumulator locality. Detail: benchmark-record |