diff --git a/.agents/issue-index.md b/.agents/issue-index.md index e0d4dae72..0615167bf 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -319,3 +319,5 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1098](https://github.com/mudler/vllm.cpp/issues/1098) | `ROAD-V1-LTX25` | `README.md` cannot be corrected, and TWO gates each refuse the fix independently. The claims that are wrong right now: **"37 registered architectures"** four times (`README.md:11,:80,:253,:296`) where `docs/FEATURES.md` says **40** in three places, corrected two commits earlier in `9143196c7`; and **ZERO `LTX` occurrences** against a `minimax` control of 7, so the video-generation announcement names one of the two shipped video families. Blocker 1: `README.md` measures **29,989 chars against `MAX_README_CHARS = 30000`** (`scripts/check-readme-structure.py:47`), so the `LTX-2.5` matrix row (~130 chars) could only land by DELETING another architecture's row - the shared-file lock AGENTS.md forbids in its own words ("Limit an entry, not a shared file"), and the third instance after the two whole-file budgets [#364](https://github.com/mudler/vllm.cpp/issues/364) retired on that argument, `MAX_CHARS` in `check-now-current.py` and the `chars` key in `check-public-doc-tables.py`. The per-ENTRY caps beside it (`MAX_CELL_CHARS = 220`, `MAX_PARAGRAPH_CHARS = 900`) are what actually stop a landing page decaying into a status log. Blocker 2, which is the decisive one: `check-doc-checkpoint.py:346-354` refuses ANY README change that does not also touch a LANDING SOURCE (`:104-113` - `.agents/mission.md`, `CMakeLists.txt`, three `benchmarks/demo/*.json`, `examples/{cli,server}/main.cpp`), evaluated PER COMMIT ([#573](https://github.com/mudler/vllm.cpp/issues/573)) so splitting the edit out does not help, and its own comment calls the rule deliberate and directly tested. It has no arm for a README CORRECTION as against README CHURN, and a correction has a natural witness: the value disagrees with the projection that owns it. So a two-family paragraph was written, MEASURED to fit at 445 chars against the old 438 with 4 to spare, and then REVERTED unlanded; it is preserved verbatim in the issue thread rather than lost. Asked: whether the checkpoint gate should distinguish correction from churn, and whether the whole-file cap should exist at all. NOT asked: raise the constant, which is widening an assertion to keep a gate green | bug | | [#595](https://github.com/mudler/vllm.cpp/issues/595) | — | `check-doc-checkpoint` keys `feature_surface` off the PATH `src/vllm/model_executor/models/`, so every edit to any model TU owes `docs/FEATURES.md` — the same classify-by-directory defect the file's own header says its rewrite removed for `src/`, `include/` and `tests/`. Measured cost: `e34d71379` (#1054) is a one-line lambda-capture change that alters no capability; the gate demanded the surface, the commit answered with prose, the prose crossed the `check-public-doc-tables` paragraph budgets, and because that checker also runs in the pre-push hook it blocked EVERY branch in the repository from pushing ([#1055](https://github.com/mudler/vllm.cpp/issues/1055), re-filed as [#1062](https://github.com/mudler/vllm.cpp/issues/1062) with a duplicate fix PR, plus [#1058](https://github.com/mudler/vllm.cpp/issues/1058) still open). The repair for the MSVC break the same commit caused ([#1068](https://github.com/mudler/vllm.cpp/issues/1068)) hit the identical demand. Narrowed here to a change in the set of `REGISTER_VLLM_MODEL(...)` registrations, which is what `check-supported-models.py` already gates the table against; adding, removing or renaming an architecture still owes the surface. The LOCK this issue names is NOT closed by that — a genuine new architecture still writes the shared table — so #595 stays open, listed under `## Owed` in [`doc-checkpoint-feature-trigger.md`](specs/doc-checkpoint-feature-trigger.md). Sibling shape for `CMakeLists.txt` -> `docs/USAGE.md` is [#515](https://github.com/mudler/vllm.cpp/issues/515) | bug | | [#1068](https://github.com/mudler/vllm.cpp/issues/1068) | `ENG-RELEASE-WINDOWS` | `main` stopped compiling under MSVC at `e34d71379` (#1054), which dropped the `[&kRequired]` capture from the `refuse` lambda in `qwen3_5_weights.cpp` as "the redundant namespace-scope capture". `kMoeExpertLayoutHelp` (`:894`) is namespace-scope and needs no capture; `kRequired` (`:929`) is a function-local `const std::string&` bound to it and IS odr-used in the lambda body, so MSVC rejects it (`error C3493`). Fixed by naming the namespace-scope constant inside the lambda, which satisfies MSVC and keeps the AppleClang diagnostic #1054 removed. It landed green because the guarding gate is a source-TEXT assertion ("rejects `[&kRequired]`, finds `[]`") that passes whether or not the TU compiles, and because `windows-msvc-*` are skipped on `main` (#503) so no baseline existed to regress. An instance of [#503](https://github.com/mudler/vllm.cpp/issues/503) | bug | +| [#1092](https://github.com/mudler/vllm.cpp/issues/1092) | `LTX25-GUIDED-VIDEO` | The LTX-2.5 **video** denoise loop runs one UNGUIDED forward per step: `Ltx2PhaseRecipe::video_guidance` is set by every recipe (`src/vllm/model_executor/models/ltx2_pipeline.cpp:1069 @ b5756ea8c`) and read by nothing, so a `pipeline_kind = one_stage` render ignores `cfg_scale = 3.0`, `stg_scale = 1.0`, `rescale_scale = 0.7` and `modality_scale = 3.0` and denoises along a different trajectory than `ti2vid_one_stage.py:221-226 @ fd4ded7f`, which builds a `FactoryGuidedDenoiser` from exactly those. `allow_guidance_override` (`ltx2_pipeline.h:534`) is dead the same way. Positive control for the grep: the same command for `audio_guidance` returns the T2A consumer at `ltx2_video.cpp:3527`. Blocks four more pipelines on one missing seam (`a2vid_two_stage.py:230`, `ti2vid_two_stages.py:248`, `ti2vid_two_stages_hq.py:271`, `keyframe_interpolation.py:232`). Spec [`ltx25-guided-video.md`](specs/ltx25-guided-video.md) | bug | +| [#1111](https://github.com/mudler/vllm.cpp/issues/1111) | `LTX25-GUIDED-VIDEO` | `Ltx2T2aGuidedStep` (`src/vllm/model_executor/models/ltx2_t2a.cpp:203-214` @ `420f6b474`) builds the STG block mask and refuses when NO bit is set, and that one predicate catches two different things. One is a request that disagrees with the CHECKPOINT - `[28]` on a two-block DiT - which is a LOCAL condition, because upstream only ever runs 48-block checkpoints and this port runs reduced ones. The other is an EMPTY list, which is upstream's OWN documented spelling for "STG off" and is refused here. Measured at Lightricks/LTX-2 `fd4ded7f`: `ltx-pipelines/docs/multimodal-guidance.md:13` says "Set to `[]` to disable STG", in the same table and the same idiom as `stg_scale` -> 0.0 and `cfg_scale` -> 1.0; `MultiModalGuiderParams.stg_blocks` DEFAULTS to `[]` (`guiders.py:204`, `field(default_factory=list)`); `--audio-stg-blocks` is `nargs="*"` (`args.py:1039-1045`, `:1107-1113`) so the empty list has a CLI spelling, and `nargs="+"` was the one-character way to forbid it; `LTX_2_3_HQ_PARAMS` SHIPS `stg_blocks=[]` on both modalities (`constants.py:105,:113`); and a whole-tree search found NO validation of `stg_blocks` anywhere - no emptiness, length or range check, no assert, no raise - with the null results recorded. `blocks=None` means EVERY block and `blocks=[]` means NO block (`perturbations.py:26-33`), so the empty list is how a caller says the second thing, and both this tree's parsers go out of their way to keep PRESENT-and-empty distinct from ABSENT - then refuse the one configuration that distinction exists to express. Upstream does not skip the pass either: `do_perturbed_generation` reads `stg_scale` alone (`guiders.py:279-281`), so an empty list disables the STG SIGNAL and not the STG COST. FOUND while repairing the fresh review of [#1092](https://github.com/mudler/vllm.cpp/issues/1092), whose B3 finding is the same divergence on the VIDEO path; that half IS fixed in flow (`ltx2_video.cpp` `ApplyGuidanceOverrides`, `ltx2_denoisers.cpp` `check_reaches_a_block`). NOT fixed here because this half landed with [#1032](https://github.com/mudler/vllm.cpp/issues/1032) and moving it changes a landed row's gated behaviour and its case "a perturbed pass over NO block is refused", which is the normal row-spec-review path rather than an in-flow repair. Until then the two arms disagree about the same request. Listed under `## Owed` in [`ltx25-guided-video.md`](specs/ltx25-guided-video.md) | bug | diff --git a/.agents/specs/ltx25-guided-video.md b/.agents/specs/ltx25-guided-video.md new file mode 100644 index 000000000..d43824805 --- /dev/null +++ b/.agents/specs/ltx25-guided-video.md @@ -0,0 +1,664 @@ +# LTX25-GUIDED-VIDEO — the guided video denoiser, as a shared seam + +Row `LTX25-GUIDED-VIDEO`, under the campaign [`ltx-2-5.md`](ltx-2-5.md). +Issue [#1092](https://github.com/mudler/vllm.cpp/issues/1092). +Base: `origin/main` @ `b5756ea8c`. +Upstream: Lightricks `LTX-2` @ `fd4ded7f` (the revision every anchor below is +read at), secondary oracle `vllm-omni` (UNPINNED, #633) for the recipe tables +this tree already mirrors. + +Paths are relative to `packages/ltx-pipelines/src/ltx_pipelines/` and +`packages/ltx-core/src/ltx_core/` in that checkout, as the rest of the LTX-2.5 +specs use them. + +## 0. Honesty statement — what this row does and does not claim + +It claims: the LTX-2.5 **video** denoise path now runs upstream's guided +denoiser, that the guidance is combined in **x0** space, that one production +pipeline (`pipeline_kind = one_stage`) reaches it on its **own default +configuration**, and that the gate can see the space error that #1039 was. + +It does not claim: a numerical comparison against a running oracle. There is +none — vLLM-Omni is UNPINNED (#633) and carries no LTX-2.5 recipe at all, and no +LTX-2.5 checkpoint on this host has a recorded sha256 (#1048). Every anchor +below is **source read** at `fd4ded7f`, and every number below is measured on +**this tree's own reduced fixture**. That is what an ungateable lane looks like +when it is stated instead of implied. + +It does not claim to retire [#1049](https://github.com/mudler/vllm.cpp/issues/1049). +See section 6c: one of that issue's four symbols is reached by this row and three +are not, and forcing the other three would mean inventing a dispatch upstream +does not have. + +## 1. Scope + +**In:** + +- `Ltx2GuidedDenoise` — `_guided_denoise` (`utils/denoisers.py:61-211`) in its + own translation unit mirroring upstream's own file. +- The four passes it assembles: `cond`, `uncond`, `ptb`, `mod`. +- The `SKIP_A2V_CROSS_ATTN` / `SKIP_V2A_CROSS_ATTN` halves of + `Ltx2DitPerturbation`, without which the `mod` pass cannot run and every video + guider default is unreachable. +- The negative conditioning for the video path — the second half of the encode + `GenerateAudioOnly` already performs and discards. +- The video guidance request extras that `default_1_stage_arg_parser` + (`utils/args.py:930-1067`) exposes, gated by `allow_guidance_override`. +- The `one_stage` pipeline as the reachable consumer. + +**Out, and owed rather than silently absent:** + +- The other video pipelines that need this seam — `a2vid_two_stage`, + `ti2vid_two_stages` ([#1093](https://github.com/mudler/vllm.cpp/issues/1093)), + `ti2vid_two_stages_hq` (owned by #921), + `keyframe_interpolation` ([#1096](https://github.com/mudler/vllm.cpp/issues/1096)), + and beside them `hdr_ic_lora` ([#1094](https://github.com/mudler/vllm.cpp/issues/1094)) + and `dubit` ([#1095](https://github.com/mudler/vllm.cpp/issues/1095)), which are + blocked on other things. This list named four pipelines and no issue numbers + when the spec was written; `281e6a120` (#1099) filed them while this branch was + open, and #1093 names this row's seam as one of the two things it waits on. + Each is its own row; this row exists so that they are ordinary porting work + rather than blocked. +- The **device-resident** arm of the `ptb` and `mod` passes. + `Ltx2DitForwardDevice` (`ltx2_device.h:136`) takes no `perturbations` + argument. Refused by name on that arm rather than run unperturbed, which would + produce a legal-looking render whose STG term is identically zero. See §4.3. +- `BatchedPerturbationConfig`'s partial blend (`attention.py:572-573`) and + batch > 1, which stay degenerate at the one batch size this port runs — the + statement `ltx2.h` already carries, unchanged. + +## 2. Upstream chain + +The executing chain for one guided step, top to bottom: + +| Step | Upstream | What it decides | +|---|---|---| +| the stage builds the model | `utils/blocks.py:480-482` — `X0Model(self._prepared_builder().build(...))` | the transformer the loop is handed is **already** an x0 model | +| the loop calls the denoiser | `utils/samplers.py:73-74` | one denoiser call per step | +| the denoiser assembles passes | `utils/denoisers.py:100-137` | `cond`, `uncond`, `ptb`, `mod`, in that order | +| the forward converts | `model/transformer/model.py:590-604` — `to_denoised(video.latent, vx, video.timesteps)` | **every** pass is x0 before any combination | +| the guider combines | `components/guiders.py:244-273`, per modality at `denoisers.py:203-204` | `cond + (cfg-1)(cond-uncond) + stg(cond-ptb) + (mod-1)(cond-modpass)`, then the rescale at `:268-271` | +| the loop post-processes | `utils/samplers.py:35` — `post_process_latent(denoised, ...)` | the conditioned tokens are pinned back **after** the guider, not per arm | +| the stepper steps | `utils/blocks.py:524-527` / `samplers.py:488-558` | Euler or ancestral, unchanged by this row | + +The pass list is **shared between the two modalities and the guiders are not**. +`denoisers.py:103-137` takes the union — one `uncond` pass if *either* guider +wants one, one `ptb` pass carrying *both* modalities' perturbations, one `mod` +pass if *either* wants one — and then `:203-204` combines each modality with its +**own** guider over the same splits. A per-modality pass list would run up to six +forwards where upstream runs four, and would give the audio stream a different +video state to cross-attend to on the video-only passes. That is the single +structural fact this port has to get right, and it is why the seam takes both +guiders rather than being called twice. + +The perturbation types are per direction, not per modality +(`guidance/perturbations.py:8-16`, applied at `model.py:443-458`): + +| Pass | Perturbations | Reaches | +|---|---|---| +| `cond` | none | — | +| `uncond` | none | negative context on both streams | +| `ptb` | `SKIP_VIDEO_SELF_ATTN` on `video_guider.stg_blocks`, `SKIP_AUDIO_SELF_ATTN` on `audio_guider.stg_blocks` | `attention.py:557` `use_attention = not all_perturbed` | +| `mod` | `SKIP_A2V_CROSS_ATTN` and `SKIP_V2A_CROSS_ATTN`, **all blocks** | `transformer.py:335,367` `cross_attn_skip_all` | + +## 3. Our baseline, derived at `b5756ea8c` + +`src/vllm/multimodal/ltx2_video.cpp:3036-3045` runs **one** forward per step and +converts its velocity: + +``` +const Ltx2DitOutputs velocity = im.on_device ? Ltx2DitForwardDevice(...) : Ltx2DitForward(...); +const std::vector v_denoised = PostProcessLatent(ToDenoised(video.latent, velocity.video, ...), video); +const std::vector a_denoised = PostProcessLatent(ToDenoised(audio.latent, velocity.audio, ...), audio); +``` + +Everything the recipe resolved for that step is set and read by nothing: + +``` +$ git grep -n 'video_guidance' -- src include # @ b5756ea8c +include/vllm/model_executor/models/ltx2_pipeline.h:526: Ltx2MultiModalGuiderParams video_guidance; +src/vllm/model_executor/models/ltx2_pipeline.cpp:1069: phase.video_guidance = params.video_guider; +``` + +The positive control for that grep is the same command for `audio_guidance`, +which returns the T2A consumer at `ltx2_video.cpp:3527`. The term and the path +set are right; the video consumer is genuinely absent. `allow_guidance_override` +(`ltx2_pipeline.h:534`) is the same shape: three recipes set it `false` and +nothing reads it. + +**What is already correct and is reused unchanged:** + +- `Ltx2MultiModalGuidance` (`ltx2_pipeline.cpp:479-522`) — `calculate` including + the unbiased-`std` rescale. Reviewed under #1032/#1039. +- `Ltx2BatchedPerturbationConfig` (`ltx2_pipeline.h:380-405`) — the full + four-type keep-mask, ported under #641 and, per #1049, constructed only by its + own test until this row. +- `ToDenoised` (`ltx2_video.cpp:277`) and `PostProcessLatent` (`:234`). +- The T2A driver (`ltx2_t2a.cpp:322-368`), which is the **template**: it is the + one place in this tree that already converts to x0 inside the model wrapper. + +**What is missing and why nothing noticed:** a token gate cannot see it, and this +path has no token gate. An unguided render returns a finite clip of the right +size, frame count and sample rate. It is #1039's family of defect one level up: +not the wrong space, the wrong number of forwards. + +## 4. Design + +### 4.1 The seam — `ltx2_denoisers.{h,cpp}` + +A new translation unit mirroring `ltx-pipelines/utils/denoisers.py`, rather than +another block inside `ltx2_pipeline.cpp`. Two reasons, and only the first is +about this row: + +1. Upstream has that file. `AGENTS.md` §Shared seams: new capability is additive + files mirroring the upstream structure. +2. `ltx2_pipeline.{h,cpp}` is concurrently edited by #921. A seam that four + later rows will extend does not want to live in the file with the most + contention. + +The transformer is a **callable**, exactly as `_guided_denoise(transformer, ...)` +takes one: + +``` +using Ltx2X0Model = std::function; +``` + +That is the structural claim this row is graded on. The x0 conversion happens +**inside the caller's lambda**, which is upstream's `X0Model` wrapper +(`blocks.py:480-482` builds it; `model.py:590-604` is its forward), so the seam +combines already-denoised tensors. Converting once after the guider instead is a +**different function on the default arm** — `rescale_scale` is 0.7 on every +video row — and that is #1039, on the audio arm, in this tree, six days ago. + +**That is caller discipline, not a type guarantee**, and this section claimed the +stronger thing ("cannot be handed a velocity") until 2026-08-17. +`Ltx2X0Outputs` carries `video_velocity` and `audio_velocity` beside `video` and +`audio` (`ltx2_denoisers.h`), because the per-arm invariant is an equation +between three tensors and cannot be checked from the denoised one alone. A lambda +that fills `video` with the velocity therefore compiles and renders. What stops +it is the gate, not the signature: mutations M1 to M4 hand the seam a velocity on +one arm each and all four are red. The claim is restated rather than the code +changed, because dropping the velocity from the struct would delete the evidence +the invariant is checked against. + +It also means the host forward and the device forward are the same seam with two +lambdas, and that the four later pipelines supply their own conditioning without +the seam knowing anything about keyframes, reference clips or two-stage +schedules. + +### 4.2 The passes + +`Ltx2GuidedDenoise` mirrors `denoisers.py:84-207` line for line: + +- `v_skip`/`a_skip` from `ShouldSkipStep` (`:84-85`); both skipping returns the + previous step's denoised pair with **no forward at all** (`:87-90`). +- `cond` always (`:100`). +- `uncond` when either guider asks or `force_uncond_pass` (`:102-109`), with the + negative context substituted per modality and `v_neg = v_context` when a + modality has none (`:107-108`). +- `ptb` when either guider perturbs, carrying both modalities' `stg_blocks` + (`:111-119`). +- `mod` when either guider isolates, all blocks, both cross directions + (`:121-137`). +- `enabled = not skip` per modality (`:158,168`), which is + `Ltx2ModalityInput::enabled` here. +- the combination per modality with that modality's own guider (`:203-204`). + +Perturbations route through `Ltx2BatchedPerturbationConfig`: one config built +over all N passes (`denoisers.py:182-187`), then `BatchSlice(i, i+1)` per pass, +then flattened into the `Ltx2DitPerturbation` the forward takes. At batch 1 the +slice is the pass's own mask, which is exactly the degeneracy `ltx2.h` already +records. + +### 4.3 The cross-attention perturbation + +`Ltx2DitPerturbation` grows two booleans, `video_cross_attn_skip_all` and +`audio_cross_attn_skip_all`, mirroring `TransformerArgs.cross_attn_skip_all` +(`transformer_args.py:70`). They gate the A2V and V2A branches at +`ltx2_dit.cpp`'s `if (run_a2v)` / `if (run_v2a)`, mirroring +`transformer.py:335` and `:367`. Note the polarity: `video.cross_attn_skip_all` +gates **A2V** (audio into video) and `audio.cross_attn_skip_all` gates **V2A**, +because the flag rides on the stream being *written*. + +The snapshot of `vx_pre`/`ax_pre` stays outside both guards, as upstream's +`vx_pre_av = vx` at `:333` does, so a build where only one direction is skipped +still reads the pre-cross state for the other. + +`ltx2.h:41-49`'s NOT-PORTED entry is corrected in the same change. Its stated +reason — "nothing upstream that this port serves constructs them" — was true for +text-to-audio, which pins `modality_scale = 1.0` (`t2a_one_stage.py:202`), +and is false for every video pipeline, all of which default it to 3.0. + +**The device arm is refused, not degraded.** `Ltx2DitForwardDevice` has no +`perturbations` parameter, so a `ptb` or `mod` pass on that arm would have to +run unperturbed. The result is a finite clip whose STG and modality terms are +identically zero — indistinguishable from a working render. The refusal names +the missing function and the owed issue. CFG alone (a different context, no +perturbation) is served on both arms. + +### 4.4 The negative conditioning + +`Generate` already encodes the positive prompt into both streams +(`ltx2_video.cpp:1771-1806`). The negative half is the same chain with +`recipe.negative_prompt` (or the `negative_prompt` extra), through the same +connector, and is encoded **only when a guider asks for it** +(`do_unconditional_generation`, `guiders.py:275-277`) — at `cfg_scale = 1.0` +there is no unconditional forward and encoding it would be a wasted host-side +12B pass per request. + +Two fallbacks exist for an engine with no text tower, matching the two that +already exist for the positive stream (`prompt_embeds_path`, +`audio_prompt_embeds_path`): `negative_prompt_embeds_path` and +`negative_audio_prompt_embeds_path`. This is a **local adaptation**, recorded as +one: upstream encodes `[prompt, negative_prompt]` in one `PromptEncoder` call +(`ti2vid_one_stage.py:166-174`) and has no embeds surface at all. The adaptation +is the existing one applied to the second of upstream's two encodings, not a new +concept. Without a tower and without those files, a guider that asks for the +unconditional pass is refused by name, exactly as T2A is at +`ltx2_video.cpp:3583-3593`. + +### 4.5 The request extras + +Mirroring `default_1_stage_arg_parser` (`utils/args.py:947-1066`), one extra per +flag, each overriding one field: + +| Extra | Upstream flag | Field | +|---|---|---| +| `video_cfg_guidance_scale` | `--video-cfg-guidance-scale` | `cfg_scale` | +| `video_stg_guidance_scale` | `--video-stg-guidance-scale` | `stg_scale` | +| `video_rescale_scale` | `--video-rescale-scale` | `rescale_scale` | +| `video_stg_blocks` | `--video-stg-blocks` | `stg_blocks` | +| `a2v_guidance_scale` | `--a2v-guidance-scale` | video `modality_scale` | +| `v2a_guidance_scale` | `--v2a-guidance-scale` | audio `modality_scale` | + +The audio row already exists for T2A (`ltx2_video.h:456-460`) and is reused for +the joint path. Every override is refused on a phase whose +`allow_guidance_override` is `false` — the distilled and retake recipes, whose +guidance is trained in — which is the first read that field has ever had. + +An extra that is PRESENT and empty is upstream's empty list for `nargs="*"`, and +stays distinct from an ABSENT extra, which takes the params table's own value. +That distinction is already made for `audio_stg_blocks` (`ltx2_video.cpp:3540`) +and is made the same way here. + +## 5. Port map + +| Upstream | Here | +|---|---| +| `utils/denoisers.py:61-211` `_guided_denoise` | `Ltx2GuidedDenoise`, `src/vllm/model_executor/models/ltx2_denoisers.cpp` | +| `utils/denoisers.py:25-28` `_POSITIVE_ONLY_GUIDER` | the default-constructed `Ltx2MultiModalGuiderParams`, whose defaults are already `cfg 1.0 / stg 0.0 / modality 1.0` | +| `model/transformer/model.py:590-604` `X0Model.forward` | the caller's `Ltx2X0Model` lambda, `ltx2_video.cpp` | +| `guidance/perturbations.py:8-16` cross types | `Ltx2DitPerturbation::{video,audio}_cross_attn_skip_all` | +| `model/transformer/transformer.py:335,367` `cross_attn_skip_all` | `ltx2_dit.cpp` A2V / V2A guards | +| `components/guiders.py:244-273` | `Ltx2MultiModalGuidance` (unchanged) | +| `utils/args.py:947-1066` | the six extras in §4.5 | +| `ti2vid_one_stage.py:211-226` | the `one_stage` consumer in `ltx2_video.cpp` | + +## 6. Gates + +``` +cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF +cmake --build build -j6 && ctest --test-dir build -j4 --output-on-failure +``` + +Focused: `ctest --test-dir build -R 'ltx2' --output-on-failure`. + +Known-red and cited by the issue that names the test, not the family: +`windows-msvc-*` (#584, no `main` baseline), `test_async_llm` (#294), +`test_engine_core_proc` (#1052), `test_serve_low_tools` (#428), +`test_cpu_x86_llamacpp_floor` exit 4 (#618). + +## 6b. Reachability — the sentence the records must carry + +Entry point: `vllm_video_generate` → `VideoEngine::Generate` +(`include/vllm.h`) on an engine loaded with `pipeline_kind = one_stage`, which +is a documented value of a documented load extra and needs no other flag. The +chain is `Generate` → the phase loop → `Ltx2GuidedDenoise`. No test constructs a +guider, a DiT or a modality by hand to reach it. + +The reachability mutation is the deletion of that call — the +`Ltx2GuidedDenoise(...)` line in the phase loop, replaced by the single unguided +forward this row removes — with the focused gate rerun. A green gate there would +mean the suite measures the seam and not the pipeline. + +## 6c. #1049 — partly retired, and the rest argued rather than deferred + +| Symbol | Before | After | +|---|---|---| +| `Ltx2BatchedPerturbationConfig` | test-only | **reached**, `ltx2_denoisers.cpp` | +| `Ltx2Guidance` | test-only | still test-only | +| `Ltx2CfgDelta` | reached only via `Ltx2Guidance` | unchanged | +| `Ltx2StgDelta` | reached only via `Ltx2Guidance` | unchanged | + +`Ltx2Guidance` is a **kind dispatch upstream does not have**. Every LTX-2 +pipeline builds a `MultiModalGuider` and calls `calculate`; there is no object +that holds a `GuiderKind` and selects between CFG-only, STG-only and multi-modal +arms. Routing the production combination through `Ltx2Guidance(kMultiModal, ...)` +to make the symbol live would add a switch statement between the caller and the +function upstream actually calls, and would still leave `Ltx2CfgDelta` and +`Ltx2StgDelta` — the two arms nothing can select — dead. #1049 stays open, its +scope narrows to those three symbols, and the honest disposition is that they +are ported-but-unreachable arms of `guiders.py:11-27,56-74`, not a wiring gap +this row can close. + +## 7. Tests to port + +Upstream's own tests for this path are `pytest` over `torch` and cannot be run +here; the harness adaptation is the whole of it. What is preserved is the +**structure of what they assert** plus the four defects this tree has already +had on the sibling arm. + +### 7.1 The per-arm invariant, on every arm + +`cond == latent - sigma*velocity`, per pass, from the trace the render records. +Exact in x0 space; in velocity space the residual is the whole sample. The RED +prints `|x0 - velocity| = 0` **exactly**, which is unambiguous. + +Non-vacuity is `REQUIRE`d, not assumed, twice: a zero latent makes the two +candidate tensors coincide, and a zero velocity on a given arm makes +`to_denoised` the identity for that arm alone. + +**Every arm**, because #1039's first gate covered only the conditional pass and +three mutations survived it. This path has four arms, so it needs four rows plus +the two double-application positions. + +### 7.2 The seam-level rescale control + +`rescale_scale = 0.0` against `0.7` on the shipped +`Ltx2MultiModalGuidance`, measuring the disagreement between combining in x0 +space and combining in velocity space. At 0.0 the linear terms are invariant and +the two are the same function; at 0.7 they are not. A gate that fires at 0.0 is +not about this defect. The existing T2A case measured 1.50e-07 against 0.352. + +**The modality term is in this control and is inert in it**, which is a weaker +statement than this section made until 2026-08-17. It said the video case "adds +the modality term, which the T2A control could not carry" and left a reader to +infer that the control therefore covers the modality arm. It does not: pinning +`modality_scale` to 1.0 moves the shipped-rescale disagreement from +`4.054e-01` to `4.118e-01`, a 1.6% change, so the fourth linear term changes +what the rescale is computed over and does not change whether the rescale is +what breaks the equivalence. Presence is coverage; it is not discriminating +power. **The modality arm's gate is the per-arm invariant in §7.1**, whose +`modality` row is what mutation M4 turns red. The two numbers are now asserted +inside the case rather than argued here, so a later reader who leans on this +control for modality coverage is contradicted by an assertion instead of by a +paragraph. + +### 7.3 The pass count + +`std(cond)/std(pred)` is 1.0 to 1e-5 on this fixture in **both** spaces, so a +naive numeric assertion on the rescale difference passes whether or not the bug +exists (7.6e-07 against a span of 3.41, measured under #1039). The instrument +that works is the **count of forwards by kind**, recorded at the call and not +asserted in prose: an arm silently skipped changes a counter no output does. + +### 7.4 The mutations this gate must survive + +| # | Mutation | Must go RED at | +|---|---|---| +| M1 | `cond` pass left in velocity space | §7.1 cond row | +| M2 | `uncond` pass left in velocity space | §7.1 uncond row | +| M3 | `ptb` pass left in velocity space | §7.1 ptb row | +| M4 | `mod` pass left in velocity space | §7.1 mod row | +| M5 | second `ToDenoised` **below** the step-0 record | the Euler-recovery check | +| M6 | second `ToDenoised` **above** the step-0 record | the guider-replay check | +| M7 | `uncond` pass given the positive context | the replay check / a uncond≠cond check | +| M8 | `mod` pass given no cross-attn perturbation | a mod≠cond check | +| M9 | `ptb` pass given no self-attn perturbation | a ptb≠cond check | +| M10 | `PostProcessLatent` applied per arm instead of after the guider | the replay check | +| M11 | the production call site deleted (reachability) | the whole case | +| M12 | the DiT ignores `video_cross_attn_skip_all` | §7.5 A2V row | +| M13 | the DiT ignores `audio_cross_attn_skip_all` | §7.5 V2A row | +| M14 | the DiT ignores BOTH cross flags | §7.5, and the shipped-path `mod != cond` check | +| M15 | the DiT SWAPS which flag gates which direction | §7.5 both rows | + +Each mutation reports three facts: `git diff --stat`, whether it **BUILT** with +the compile-error count, and the exit code captured directly. A non-building +mutation reads exactly like a passing test. + +### 7.5 The two cross directions, gated per direction + +M12, M13 and M15 were all **GREEN** against the first draft of this row's gate, +which had no direct DiT-level cross case: only M14 was caught. A build that +plumbs both flags and applies exactly one, or applies both to the wrong +directions, renders — on the DEFAULT video arm, whose `modality_scale` is 3.0 — +with the isolated-modality term half wrong. + +Two things made the shipped-path case blind to it. The end-to-end +`MaxAbsDiffOf(video_first_modality, video_first_cond)` still fires with one +direction applied, because the modality pass still differs from `cond`. And +`Ltx2ConditioningTrace::video_modality_skipped_{a2v,v2a}` is assigned from the +perturbation struct **the seam built** (`ltx2_denoisers.cpp:315-316`), so it +records what was handed over and nothing about what the DiT did with it — while +its message claimed the latter. + +The instrument is a **direct `Ltx2DitForward` case per direction**, mirroring the +self-attention one this file already had. Separation comes from upstream's own +predicates (`transformer.py:265-269`): `run_a2v` needs the VIDEO stream enabled +and the audio stream merely PRESENT, and `run_v2a` the reverse. So a forward with +`audio->enabled = false` runs A2V alone, and one with `video->enabled = false` +runs V2A alone. Each row asserts both halves — the flag for that direction MOVES +the written stream, and the flag for the other direction leaves it BIT-IDENTICAL +— which is what makes the swap detectable rather than only the omission. The +per-block-pair coupling that defeats a both-enabled forward (block 1's V2A reads +what block 0's A2V wrote) never arises, because only one direction runs at all. + +## 8. Risks and decisions + +**R1 — the space error, on the default arm.** `rescale_scale` is 0.7 on the +2.4/2.5 video row and 0.45 on the HQ row; both are non-zero, so a space error +hits the default. Mitigated by §7.1 and §7.2, and by the conversion living in +the caller's lambda where the seam cannot receive a velocity. + +**R2 — the shared pass list.** Assembling per modality is the plausible wrong +design and it renders. Mitigated by the pass-count trace and by the seam taking +both guiders. + +**R3 — cost.** Four forwards per step where there was one. That is upstream's +own cost — `denoisers.py` batches them into one call, this port runs them +serially — and it is a **correctness** row, so the throughput axis is not traded +against it. Recorded, not hidden: a `one_stage` render is now up to 4x the DiT +work per step. `distilled_two_stage`, the default recipe and the one every +benchmark on this row's campaign used, denoises with `SimpleDenoiser` upstream +(`distilled.py:266,295`) and is **unchanged** by this row. + +**R4 — the device arm.** §4.3. Refused by name, owed by a new issue, rather than +run unperturbed. + +**R5 — concurrent edits.** #921 touches `ltx2_pipeline.{h,cpp}` and the stepper +enum. This row's new code is in a new file; its edits to `ltx2_pipeline.h` are +additive constants only. + +## 9. Stop conditions + +Stop and report `NEEDS_DECISION` rather than narrowing silently if: + +- the `mod` pass cannot be made to differ from the `cond` pass on the fixture, + because then §7.4 M8 cannot go red and the isolated-modality arm is gated by + nothing; +- the guided `one_stage` render cannot be reached without a text tower **and** + the negative-embeds adaptation is judged out of scope, because then the + consumer is unreachable in-tree and this becomes a seam-only row. + +## Owed + +- **The device-resident `ptb` and `mod` passes.** `Ltx2DitForwardDevice` takes + no `perturbations`. Owned by this row's follow-up issue; refused by name until + then. +- **The other four pipelines** — `a2vid_two_stage`, `ti2vid_two_stages` + ([#1093](https://github.com/mudler/vllm.cpp/issues/1093)), + `ti2vid_two_stages_hq` (#921), + `keyframe_interpolation` ([#1096](https://github.com/mudler/vllm.cpp/issues/1096)). + Each needs its own row; none is blocked on this seam any more. +- **#1049's remaining three symbols** — see §6c. +- **[#1111](https://github.com/mudler/vllm.cpp/issues/1111) — the T2A arm still + refuses an EMPTY `audio_stg_blocks`.** The same divergence this row's review + found on the video path, in `ltx2_t2a.cpp:203-214`, which landed with #1032 and + is not this row's code. Fixing it moves a landed row's gated behaviour and one + of its cases, so it takes the normal row-spec-review path. Until then the two + arms disagree about the same request. +- **An oracle-run comparison.** vLLM-Omni is UNPINNED (#633) and carries no + LTX-2.5 recipe; no LTX-2.5 checkpoint here has a recorded sha256 (#1048). The + guidance arithmetic is gated against upstream **source**, not against upstream + **output**, and that is the ceiling on this row's evidence. + +## 10. What the implementation actually did, against §4 + +Three things moved from the design, and each is recorded here rather than left +for a reader to diff. + +**The seam runs on EVERY video render, not only the guided ones.** §4 left this +open. The implementation routes the phase loop through `Ltx2GuidedDenoise` +unconditionally, because a phase whose recipe sets no guidance keeps +`Ltx2MultiModalGuiderParams`'s own defaults and those ARE +`_POSITIVE_ONLY_GUIDER` (`denoisers.py:25-28`): one pass, and a `calculate` whose +every term is zero. Upstream selects `SimpleDenoiser` by PIPELINE +(`distilled.py:266,295`) rather than by params, and the two agree here because +the recipes that select it are exactly the recipes whose guidance is the no-op +one. **Measured, not argued:** every existing golden in `test_ltx2_video` held +unchanged across the change, including the `distilled_two_stage` renders and the +DFR and retake ones. + +**One refusal that upstream does not have.** It is a case where upstream's +behaviour is a silent zero rather than an error, and where the render finishes: + +- an `stg_blocks` list naming no block this checkpoint has. Upstream's + `Perturbation.is_perturbed` is a membership test (`perturbations.py:26-33`), so + `[28]` on a two-block DiT perturbs nothing and leaves + `stg_scale * (cond - perturbed)` at exactly zero. Upstream never meets it + because it only runs 48-block checkpoints; this port runs reduced ones. The + refusal is about a request that disagrees with the **checkpoint**, which is a + local condition rather than an upstream one. + +**A second refusal was landed on this branch and is retired in the review +repair.** An EMPTY `stg_blocks` beside a non-zero STG scale was refused for the +same "the term would be exactly zero" reason. Every clause of that reason is +true, and none of it makes the configuration illegal upstream — which is the only +question a mirror gets to ask. Measured at `fd4ded7f`: + +| Evidence | Where | +|---|---| +| "Set to `[]` to disable STG", in the same table and idiom as `stg_scale` → 0.0 | `ltx-pipelines/docs/multimodal-guidance.md:13` | +| `MultiModalGuiderParams.stg_blocks` DEFAULTS to `[]` | `guiders.py:204` | +| `--video-stg-blocks` / `--audio-stg-blocks` are `nargs="*"`, so `[]` has a CLI spelling; `nargs="+"` was the one-character way to forbid it | `args.py:979-985`, `:1039-1045`, `:1107-1113` | +| `LTX_2_3_HQ_PARAMS` SHIPS `stg_blocks=[]` on both modalities | `constants.py:105`, `:113` | +| no validation of `stg_blocks` anywhere in that tree — no emptiness, length or range check | measured by a whole-tree search with the null results recorded | + +`blocks=None` means EVERY block and `blocks=[]` means NO block +(`perturbations.py:26-33`), so the empty list is how a caller says the second +thing — and `ApplyStgBlocksExtra` exists precisely to keep PRESENT-and-empty +distinct from ABSENT. Refusing it made that distinction unreachable, which is +the shape of the defect rather than a matter of taste. Upstream does not skip +the pass either: `do_perturbed_generation` reads `stg_scale` alone +(`guiders.py:279-281`), so the `ptb` entry is appended and its result equals +`cond`. This port now does the same, and the case asserts the exact equality +rather than a tolerance. + +**The same refusal exists on the T2A arm and is NOT changed here.** +`ltx2_t2a.cpp:203-214` computes the block mask and refuses when no bit is set, +which catches the empty list as well as the out-of-range one. It landed with +[#1032](https://github.com/mudler/vllm.cpp/issues/1032) and is not this row's +code; changing it would alter a landed row's gated behaviour and one of its +cases. Filed and listed under `## Owed`. + +**A cross perturbation that differs between blocks is refused rather than +widened.** `Ltx2DitPerturbation` carries one boolean per direction, which is +sound only while the config says the same thing on every block — which +`blocks=None` guarantees. That is checked in the seam rather than assumed. + +## 11. Outcome + +Landed on `row/LTX25-GUIDED-VIDEO`. Twelve mutations, eleven RED and one proven +an IDENTITY; the table with all three facts per mutation is in the pull request +body. + +**The one green, and what chasing it changed.** `post_process_latent` ADDED per +arm, with the after-guider application left in place, runs the whole suite to +71 cases / 2145 assertions / exit 0. Two readings of that green were tried and +the first one was wrong. + +The first reading was that the gate could not see it, so a check was added that +replays the guider over arms REBUILT from `latent` and the four raw velocities -- +independent of anything applied to the arms alike. It did not move the green +either, which was the clue. + +The second reading, measured rather than argued, is that the mutation is a NO-OP. +`post_process_latent` is `x*mask + clean*(1-mask)`, so it can only touch a token +whose denoise mask is 0; such a token's per-token sigma is 0 +(`timesteps_from_mask`, `utils/helpers.py:494-503`), so `X0Model` returns +`latent - 0*v`, which is `latent`; and a conditioned token's `latent` IS its +clean value. Every arm already equals what post-processing would write. + +**What that produced is a better test, not a waived mutation.** The tempting +proof of the above -- "so applying it anywhere is the same" -- is false for the +guider's RESULT, because the rescale (`guiders.py:268-271`) is a scalar over the +whole tensor and multiplies the conditioned tokens too. Take the after-guider +application away, or move it down into the denoiser, and those tokens leave the +step scaled by a number nobody asked for. The row therefore carries a CONDITIONED +`one_stage` case which asserts both halves: that no arm was touched, and that the +guider's result was, on exactly the mask-0 tokens. The MOVE mutation is RED +against it (71 cases / 2133 assertions / exit 1); the ADD mutation stays green +and the case now says why in an assertion instead of leaving it unexplained. + +**A residual, stated.** Nothing here compares against a running oracle; see §0. + +### 11.1 The fresh review, and what it moved + +One BLOCKING finding and ten non-blocking ones. The core was reproduced and is +not revisited: the seam, the x0 space, all four arms, the reachability, and the +gate. The reviewer also confirmed the live defect independently — `video_guidance` +had exactly two hits at the merge base, a declaration and a write with no reader, +against a positive control where `audio_guidance` finds its T2A consumer. + +| Finding | Disposition | +|---|---| +| B1 (blocking) — the two cross booleans were not gated per direction: M12, M13 and M15 were GREEN | REPAIRED. §7.5, a direct DiT-level case per direction. All four mutations now RED | +| B2 — `docs/FEATURES.md` said T2A was "the only GUIDED arm", which this row's own new row made false | REPAIRED in the existing cell, 202 of 220 chars, prose paragraphs unchanged at 21 of 21 | +| B3 — the EMPTY `stg_blocks` refusal diverges from upstream | REPAIRED on this row's paths; the T2A one is #1111. See §10 | +| B4 — `INFO("arm = " << arm.name)` printed `arm = 1` | REPAIRED at all three sites in the file | +| B5 — the rescale control's modality claim was numerically inert | RESTATED, and now asserted rather than argued. See §7.2 | +| B6 — upstream anchor drift | REPAIRED, 41 scripted replacements with the hit count asserted per edit. See §11.2 | +| B7 — "the seam cannot be handed a velocity" is caller discipline, not a type guarantee | RESTATED in §4.1 and in `ltx2_denoisers.h`; the code is unchanged, which is the right outcome | +| B8 — the branch was behind `origin/main` | MERGED and re-gated; §1 now carries #1093 to #1096 from `281e6a120` | +| B9, B11 | recorded by the reviewer as not this repair's | +| B10 — the new `docs/USAGE.md` section documented flags and not extra keys | REPAIRED: the `/v1/videos` caveat, a flag-to-extra table, and the empty-list behaviour | + +**One sub-claim was REJECTED on evidence.** B3 argued an asymmetry: that +`audio_stg_blocks=""` is still accepted on `t2a_one_stage`, because that path +returns before `ApplyGuidanceOverrides`. It does return there, and the request is +still refused — by `ltx2_t2a.cpp:203-214`, which builds the block mask and fails +when no bit is set, and which `git log -S` puts on `main` at `0b0b8900f` with +#1032 rather than on this branch. So there is no asymmetry today; the two arms +agree, and both diverge from upstream. Fixing the video half creates the +asymmetry, which is why #1111 exists and is listed under `## Owed` rather than +left implied. + +### 11.2 The anchors + +Re-derived against `fd4ded7f` from the sentence making each claim, never by +reading text out of the cited span. Corrected: `_guided_denoise` **61-211** (was +62-207); `enabled = not skip` at **158, 168** (was 151, 161 — the `= None` +initializers); the V2A `cross_attn_skip_all` guard at **367** (366 is blank); the +batched config built at **182-187** (172-176 is the comment plus the per-sample +replication at :175); the partial blend at **572-573**; the one `PromptEncoder` +call at **166-174**; `default_1_stage_arg_parser` **930-1067** and its guider +flags **947-1066**; the two `--*-stg-blocks` flags at **979-985** and +**1039-1045**; `cross_attn_skip_all` DECLARED at `transformer_args.py:70` (118 is +a call site); `modality_scale = 3.0` at `constants.py:54, :64` with +`_PARAMS_SINCE_VERSION` at **130-133** (the cited 40-80 covers neither); +`CFGGuider` / `STGGuider` at **11-27** and **56-74**; the `perturbations` +ARGUMENT at `model.py:493` (492 is the `def`); `t2a_one_stage.py:202` (200-201 +are its comment); `ti2vid_one_stage.py:211-226` (210 is blank). + +No gate protects a spec anchor ([#632](https://github.com/mudler/vllm.cpp/issues/632)), +so the edits were applied by a script that asserts the expected hit count per +replacement and refuses the whole run on a mismatch — two of the 43 planned edits +were caught that way and re-derived. + +## Now + +The fresh review returned CHANGES REQUESTED; the blocking finding and the nine +non-blocking ones this repair owns are answered on the branch (§11.1), the branch +is merged up to `origin/main`, and the row awaits a second fresh review. diff --git a/CMakeLists.txt b/CMakeLists.txt index d9b5686b2..898640405 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -879,6 +879,12 @@ add_library(vllm STATIC # runs the DiT with `video = nullptr`, and keeping that call in one file is # what makes "does anything else pass a video stream" a readable question. src/vllm/model_executor/models/ltx2_t2a.cpp + # LTX-2.5 (ROW LTX25-GUIDED-VIDEO, issue #1092): `_guided_denoise` — the pass + # assembly four unported pipelines are each blocked on, and the piece the + # VIDEO denoise loop never had. Its own TU because upstream has its own file + # (ltx-pipelines utils/denoisers.py) and because ltx2_pipeline.cpp, where the + # guiders live, is a concurrent-edit lock. + src/vllm/model_executor/models/ltx2_denoisers.cpp # LTX-2.5 (ROW MODEL-DIFFUSION-LTX25, phase L6): the quantized loaders — the # FP8 and torchao-NVFP4 DiT arms, the torchao-NVFP4 text encoder, and the # load-time device staging GB10's ATS penalty makes the default. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index d3bf3974b..d8d566ec9 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -173,8 +173,11 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | LTX-2.5 Conv VAE decode arithmetic width | LTX-2.5 video VAE | `test_ltx2_vae` "the decode's convolution accumulates in f32", entering through `Ltx2VideoDecodeStreaming`; widening the accumulator to `double`, or deleting the production call site, each turns it RED | **f32**, the width `F.conv3d` uses at f32 AND bf16 (MEASURED). Was f64 at 8 sites ([#1008](https://github.com/mudler/vllm.cpp/issues/1008)). Conv sums BLOCKED per input channel, as torch's. STORAGE stays f32; bf16 owed | | LTX-2.5 Conv VAE decode threading | LTX-2.5 video VAE | `test_ltx2_vae` "the decode DISPATCHES its convolutions to the CPU threadpool" and "...BIT-IDENTICAL across thread counts", through `Ltx2VideoDecodeStreaming`; 34 golden margins UNCHANGED; TSan clean | **Parallel** over CONV output lines via `vt::cpu::ParallelForRows` ([#1009](https://github.com/mudler/vllm.cpp/issues/1009)). ~9x at 16-20 workers, contended box, 21-23% spread. Bit-identical at any count | | LTX-2.5 retake (`RetakePipeline`, regenerate a time window) | LTX-2.5 DiT + video VAE encoder | `test_ltx2_retake` 4/4 (69 assertions) and 4 `test_ltx2_video` cases entering through `Generate`; mask, conform and the four-way plan pinned to upstream `fd4ded7f` | `--pipeline-kind retake` on `ltx2-gen`. Source is a `frame_%06d.ppm` DIRECTORY; a container is REFUSED (no demuxer). Geometry comes from the clip. A folder has no audio, so the soundtrack is generated | -| LTX-2.5 text-to-audio (`T2AOneStagePipeline`) | LTX-2.5 DiT + audio VAE, no video VAE | `test_ltx2_video`'s `ltx2 t2a:` cases, entering through `Generate`; 18 mutations, 17 DETECTED (four by review of a conditional-only #1039 gate) and the 18th proven an identity, not a blind spot | `--pipeline-kind t2a_one_stage`. NO picture: 0 frames, no mux argv. The only GUIDED arm (CFG + STG, 3 forwards/step), so it needs a text tower. CPU only; the device forward is refused by name | +| LTX-2.5 text-to-audio (`T2AOneStagePipeline`) | LTX-2.5 DiT + audio VAE, no video VAE | `test_ltx2_video`'s `ltx2 t2a:` cases, entering through `Generate`; 18 mutations, 17 DETECTED (four by review of a conditional-only #1039 gate) and the 18th proven an identity, not a blind spot | `--pipeline-kind t2a_one_stage`. NO picture: 0 frames, no mux argv. The only AUDIO-ONLY guided arm (CFG + STG, 3 forwards/step), so it needs a text tower. CPU only; the device forward is refused by name | | LTX-2.5 T2A guidance space | LTX-2.5 DiT (T2A arm) | `test_ltx2_video` "the guider is handed x0 predictions" through `Generate`, on all 3 arms plus the guider output and the Euler input; a seam case puts the two spaces 1.5e-07 apart at rescale 0 and 0.352 at 0.7 | Combines **denoised (x0)**, mirroring `X0Model` (`model.py:590-604`). Was velocity space, which agrees only at rescale 0 ([#1039](https://github.com/mudler/vllm.cpp/issues/1039)) | +| LTX-2.5 VIDEO guidance | LTX-2.5 DiT, joint video+audio | `test_ltx2_video`'s `ltx2 one_stage:` cases through `Generate`; all FOUR arms carry the x0 invariant and the guider output replays EXACTLY | `--pipeline-kind one_stage` runs `_guided_denoise`: 4 forwards/step, combined per modality in **x0**. Was ONE unguided forward, every `video_guidance` field dead ([#1092](https://github.com/mudler/vllm.cpp/issues/1092)) | +| LTX-2.5 cross-attention perturbations | LTX-2.5 DiT | `test_ltx2_video` gates each direction ALONE, on a forward where one stream is PRESENT but DISABLED so only that one runs: the flag moves the stream it writes, the other leaves it bit-identical. Swapping the two is RED | `SKIP_A2V_CROSS_ATTN` / `SKIP_V2A_CROSS_ATTN` ported, which `modality_scale = 3.0` selects on every video row. The DEVICE forward takes no perturbations, so that pass is refused there by name | +| LTX-2.5 guidance knobs | LTX-2.5 request surface | `test_ltx2_video` renders with an override and refuses one on a fixed recipe | Seven video/audio guider extras mirroring `default_1_stage_arg_parser`, plus a negative embeds pair for a tower-less engine. Refused whole on `distilled_two_stage` and `retake`, whose guidance is distilled in | | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) | | MTP speculation DEPTH (`num_speculative_tokens` > 1) | Qwen3.5/3.6 `mtp.*` heads | k=1..4 through the loader, greedy tokens unmoved, two witnesses per arm: the draft decode forwards the propose RAN, and whether the DELIVERED draft row varied with depth. `test_mtp_depth` 5/5, 63 assertions | Default stays k=1. NO speed claim at k>1. Drafts are proposed and verified, never ACCEPTED, and neither witness proves per-column provenance. Both await the owed DGX gate (#81) | | DFlash block-diffusion | Qwen3 (DFlash draft) | near-tie e2e 27/27 vs vLLM | 2.9x over spec-off, 1.003x vs vLLM DFlash-on | diff --git a/docs/USAGE.md b/docs/USAGE.md index 7929c6bd5..3579e30b0 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -941,14 +941,18 @@ ignored — upstream passes a 512x512 placeholder whose height and width it documents as unused, and only the frame count and the recipe's frame rate are read, to derive the duration. -**It is the only GUIDED arm, and that changes what it costs and what it needs.** -The distilled video recipes run one DiT forward per step. This one runs -**three** by default — conditional, unconditional, and one with the audio +**It is a GUIDED arm, and that changes what it costs and what it needs.** The +distilled video recipes run one DiT forward per step. This one runs **three** by +default — conditional, unconditional, and one with the audio self-attention perturbed (STG) — so it is roughly 3x the work per step, and it **requires a text tower**, because the unconditional pass conditions on the negative prompt. Loading with `prompt_embeds_path` alone gets a refusal naming `--audio-cfg-guidance-scale 1.0` as the way to turn the unconditional pass off. +It was the only guided arm here until row LTX25-GUIDED-VIDEO +([#1092](https://github.com/mudler/vllm.cpp/issues/1092)) gave the joint video +path its own denoiser; see *LTX-2.5 video guidance* below. + Six per-generation knobs mirror upstream's own CLI, and each takes the checkpoint generation's value when absent: `--negative-prompt`, `--audio-cfg-guidance-scale` (7.0), `--audio-stg-guidance-scale` (1.0), @@ -975,6 +979,99 @@ at the recipe's own guider values. pipeline: the device forward takes both streams by reference and this pipeline has no video stream to give it. Use `--device cpu`. +### LTX-2.5 video guidance: `--pipeline-kind one_stage` + +`one_stage` mirrors upstream's `TI2VidOneStagePipeline`, which builds a +`FactoryGuidedDenoiser` from the params table's own video and audio guiders. On +the 2.4/2.5 lineage those resolve to `cfg_scale = 3.0`, `stg_scale = 1.0`, +`rescale_scale = 0.7` and `modality_scale = 3.0`. + +Until [#1092](https://github.com/mudler/vllm.cpp/issues/1092) this port read none +of it: the joint denoise loop ran one unguided forward per step. A `one_stage` +render therefore finished, at the right size and frame count, along a different +trajectory than upstream's. It now runs **four** forwards per step and combines +them per modality: + +| Pass | What differs | Selected by | +|---|---|---| +| conditional | nothing | always | +| unconditional | the negative conditioning | `cfg_scale != 1.0` | +| perturbed | video/audio self-attention skipped on `stg_blocks` | `stg_scale != 0.0` | +| isolated modality | the audio<->video cross attention off in every block | `modality_scale != 1.0` | + +Seven per-generation knobs mirror upstream's `default_1_stage_arg_parser` and +each takes the checkpoint generation's value when absent. The audio row and +`--negative-prompt` are shared with text-to-audio and are no longer refused on a +video pipeline; upstream's parser carries both rows side by side, and the old +refusal rested on a reading of upstream that was wrong and harmless only while +nothing here read them. + +| `ltx2-gen` flag | per-generation extra | meaning | +|---|---|---| +| `--video-cfg-guidance-scale` | `video_cfg_guidance_scale` | video `cfg_scale`; `1.0` turns the unconditional forward off | +| `--video-stg-guidance-scale` | `video_stg_guidance_scale` | video `stg_scale`; `0.0` turns the perturbed forward off | +| `--video-rescale-scale` | `video_rescale_scale` | video `rescale_scale`, applied to the DENOISED prediction | +| `--video-skip-step` | `video_skip_step` | `0` never skips; `n` runs every `n+1`-th step | +| `--video-stg-blocks` | `video_stg_blocks` | comma separated block indices; EMPTY disables STG, see below | +| `--a2v-guidance-scale` | `a2v_guidance_scale` | video `modality_scale`; `1.0` turns the isolated-modality forward off | +| `--v2a-guidance-scale` | `v2a_guidance_scale` | audio `modality_scale` | +| `--negative-prompt` | `negative_prompt` | the unconditional forward's conditioning | + +The audio row is the same six spellings with `audio_` in place of `video_`: +`audio_cfg_guidance_scale`, `audio_stg_guidance_scale`, `audio_rescale_scale`, +`audio_skip_step`, `audio_stg_blocks`, and `v2a_guidance_scale` for its +`modality_scale`. + +Those extras ride the per-generation `extra_keys` / `extra_values` array on +`vllm_video_params`, so the C ABI reaches the same path with no new field. They +are per-GENERATION and therefore reach the CLI and the C ABI and **not** +`/v1/videos`, which forwards no per-generation extra to any engine +([#928](https://github.com/mudler/vllm.cpp/issues/928)). `pipeline_kind` is a +LOAD knob and does reach the server, so a server started with +`--video-extra pipeline_kind=one_stage` renders every request through the guided +denoiser at the recipe's own guider values and no request can change them. + +**An EMPTY `--video-stg-blocks` is accepted and means "perturb no block".** That +is upstream's own idiom — `docs/multimodal-guidance.md:13` says "Set to `[]` to +disable STG", the field defaults to `[]`, the flags are `nargs="*"`, and the +shipped HQ params row uses it — and it stays distinct from OMITTING the flag, +which takes the params table's value. It disables the STG signal and not the STG +cost: upstream selects the perturbed pass from `stg_scale` alone, so the forward +still runs and contributes exactly zero. Set the scale to `0.0` to skip the +forward as well. This page and this port refused the empty list until +2026-08-17. + +**The unconditional forward needs a negative conditioning, and there are two +ways to supply one.** With a text tower, `--negative-prompt` (or the recipe's +own default) is encoded through the same chain as the positive prompt. Without +one, `--negative-prompt-embeds` and `--negative-audio-prompt-embeds` — the LOAD +extras `negative_prompt_embeds_path` and `negative_audio_prompt_embeds_path` — +are the negative half of the `prompt_embeds_path` fallback: two files at the +DiT's two cross-attention widths, the same row count as the positive pair. Being +LOAD extras they DO reach the server, through `--video-extra`. With neither, a +`cfg_scale` other than 1.0 is **refused by name** rather than served the positive +context twice, which would leave the whole classifier-free term at exactly zero. + +**A block index the checkpoint does not have is refused**, which is the case the +empty list above is NOT. `stg_blocks` is a membership test upstream, so naming +block 28 on a model with fewer blocks perturbs nothing and leaves +`stg_scale * (cond - perturbed)` at exactly zero — the same zero, reached by a +request that disagrees with the checkpoint rather than by a caller who asked for +no perturbation. Upstream never meets it because it only ships 48-block +checkpoints, so this refusal is local to this port and is named as such. + +**The distilled and retake recipes refuse every one of these flags.** Their +guidance is distilled into the weights, so honouring an override would sample a +trajectory the weights were never trained for. Their guiders are upstream's +positive-only one, so they still issue one forward per step and their output is +unchanged by this row. + +**The accelerator is refused for the perturbed and isolated-modality passes.** +`Ltx2DitForwardDevice` takes no perturbation argument, so those two passes on +`device = 1` would run an unperturbed forward and leave both terms at zero. +Classifier-free guidance alone is a different context and no perturbation, and +runs on both arms. + **What is not served.** `temporal_upsample_rounds` is defined and refused above `0`: the rounds loop that temporally doubles the latent, re-tiles the canvas and stitches it back is not ported. The refusal names it, and it names three things @@ -2540,6 +2637,8 @@ seam's `prompt_embeds_path`, which carries the video stream), `pipeline_kind` (default `distilled_two_stage`; also `one_stage`, `dmd2`, `dfr`, `retake` and `t2a_one_stage`), `model_version` (only for a checkpoint that declares none), `dit_config_path`, `encoder_config_path`, +`negative_prompt_embeds_path` and `negative_audio_prompt_embeds_path` (the +negative half of the same fallback, for the unconditional forward), `allow_unported_modules`, `max_phase`, `prompt_embeds_valid_rows`, `upsampler_path`, `duration_head_path`, `lora_path` and `lora_strength` — twelve keys, which is `kKnownLoadExtras` (`ltx2_video.cpp:377-383`) in order. The two diff --git a/examples/ltx2_gen/main.cpp b/examples/ltx2_gen/main.cpp index 7f4106f49..f94908d04 100644 --- a/examples/ltx2_gen/main.cpp +++ b/examples/ltx2_gen/main.cpp @@ -183,7 +183,19 @@ const char* Need(int argc, char** argv, int i, const char* flag) { "--audio-skip-step and --audio-stg-blocks are upstream\'s own flags; absent, each\n" "takes the checkpoint generation\'s own value. --audio-stg-blocks is comma\n" "separated and a block index outside the DiT\'s layer count is refused rather\n" - "than clamped. The accelerator is REFUSED by name on this pipeline.\n"); + "than clamped. The accelerator is REFUSED by name on this pipeline.\n\n" + "GUIDANCE ON A VIDEO RENDER. --pipeline-kind one_stage runs upstream's guided\n" + "denoiser: FOUR DiT forwards per step (conditional, unconditional, perturbed,\n" + "and one with the audio<->video cross attention off), combined per modality in\n" + "x0 space. --video-cfg-guidance-scale, --video-stg-guidance-scale,\n" + "--video-rescale-scale, --video-skip-step, --video-stg-blocks,\n" + "--a2v-guidance-scale and --v2a-guidance-scale are upstream's own flags and\n" + "take the checkpoint generation's value when absent. The unconditional forward\n" + "needs a NEGATIVE conditioning: either a text tower plus --negative-prompt, or\n" + "--negative-prompt-embeds with --negative-audio-prompt-embeds. Absent both, a\n" + "cfg scale other than 1.0 is refused by name. --pipeline-kind\n" + "distilled_two_stage and retake distil their guidance INTO the weights and\n" + "refuse every one of these flags rather than applying it.\n"); std::exit(code); } @@ -205,6 +217,13 @@ int main(int argc, char** argv) { std::string negative_prompt, audio_cfg_scale, audio_stg_scale, audio_rescale; std::string audio_skip_step, audio_stg_blocks; + // THE VIDEO GUIDER (row LTX25-GUIDED-VIDEO, #1092): the other half of the same + // parser, `default_1_stage_arg_parser` (utils/args.py:947-1066). `--negative- + // prompt` above is shared by both, which is why it is not repeated here. + std::string video_cfg_scale, video_stg_scale, video_rescale, video_skip_step; + std::string video_stg_blocks, a2v_scale, v2a_scale; + std::string negative_embeds, negative_audio_embeds; + // The extras are BORROWED by the load call, so the strings must outlive it. // Kept as two parallel vectors of owned strings plus the char* views the ABI // takes, built once after parsing. @@ -238,6 +257,13 @@ int main(int argc, char** argv) { else if (f == "--model-version") SetExtra("model_version", Need(argc, argv, ++i, f.c_str())); else if (f == "--pipeline-kind") SetExtra("pipeline_kind", Need(argc, argv, ++i, f.c_str())); else if (f == "--upsampler") SetExtra("upsampler_path", Need(argc, argv, ++i, f.c_str())); + else if (f == "--negative-prompt-embeds") { + negative_embeds = Need(argc, argv, ++i, f.c_str()); + SetExtra("negative_prompt_embeds_path", negative_embeds); + } else if (f == "--negative-audio-prompt-embeds") { + negative_audio_embeds = Need(argc, argv, ++i, f.c_str()); + SetExtra("negative_audio_prompt_embeds_path", negative_audio_embeds); + } // Kept although the library REFUSES this extra by name (#611): the duration // head is unported, and forwarding the flag gets the caller that named // refusal instead of "unknown option", which says nothing about why. @@ -295,6 +321,20 @@ int main(int argc, char** argv) { audio_skip_step = Need(argc, argv, ++i, "--audio-skip-step"); else if (f == "--audio-stg-blocks") audio_stg_blocks = Need(argc, argv, ++i, "--audio-stg-blocks"); + else if (f == "--video-cfg-guidance-scale") + video_cfg_scale = Need(argc, argv, ++i, "--video-cfg-guidance-scale"); + else if (f == "--video-stg-guidance-scale") + video_stg_scale = Need(argc, argv, ++i, "--video-stg-guidance-scale"); + else if (f == "--video-rescale-scale") + video_rescale = Need(argc, argv, ++i, "--video-rescale-scale"); + else if (f == "--video-skip-step") + video_skip_step = Need(argc, argv, ++i, "--video-skip-step"); + else if (f == "--video-stg-blocks") + video_stg_blocks = Need(argc, argv, ++i, "--video-stg-blocks"); + else if (f == "--a2v-guidance-scale") + a2v_scale = Need(argc, argv, ++i, "--a2v-guidance-scale"); + else if (f == "--v2a-guidance-scale") + v2a_scale = Need(argc, argv, ++i, "--v2a-guidance-scale"); else if (f == "--regenerate-video") regen_video = Need(argc, argv, ++i, "--regenerate-video"); else if (f == "--regenerate-audio") @@ -376,7 +416,17 @@ int main(int argc, char** argv) { std::make_pair("audio_stg_guidance_scale", &audio_stg_scale), std::make_pair("audio_rescale_scale", &audio_rescale), std::make_pair("audio_skip_step", &audio_skip_step), - std::make_pair("audio_stg_blocks", &audio_stg_blocks)}) { + std::make_pair("audio_stg_blocks", &audio_stg_blocks), + // THE VIDEO GUIDER (#1092). Per-generation for the same + // reason the audio row is, and refused whole on a recipe + // whose guidance is distilled into the weights. + std::make_pair("video_cfg_guidance_scale", &video_cfg_scale), + std::make_pair("video_stg_guidance_scale", &video_stg_scale), + std::make_pair("video_rescale_scale", &video_rescale), + std::make_pair("video_skip_step", &video_skip_step), + std::make_pair("video_stg_blocks", &video_stg_blocks), + std::make_pair("a2v_guidance_scale", &a2v_scale), + std::make_pair("v2a_guidance_scale", &v2a_scale)}) { if (kv.second->empty()) continue; gen_keys.emplace_back(kv.first); gen_values.push_back(*kv.second); diff --git a/include/vllm/model_executor/models/ltx2.h b/include/vllm/model_executor/models/ltx2.h index 998cda7c9..472e1c080 100644 --- a/include/vllm/model_executor/models/ltx2.h +++ b/include/vllm/model_executor/models/ltx2.h @@ -39,26 +39,35 @@ // message naming the missing phase rather than silently computing in f32. // // NOT PORTED IN L2, recorded here so it cannot be discovered later: -// - The CROSS-attention guidance perturbations only, as of row -// LTX25-T2A-ONE-STAGE (#1005): `SKIP_A2V_CROSS_ATTN` and -// `SKIP_V2A_CROSS_ATTN` (guidance/perturbations.py:8-16, -// transformer.py:330-397 `cross_attn_skip_all`). Nothing upstream that this -// port serves constructs them — STG is built from `stg_blocks` and reaches -// the SELF-attention types alone (utils/constants.py:49-68 through -// guiders.py:194-211) — and `Ltx2Attention` refuses the flag on a cross call -// by name rather than applying the self-attention rule to it. +// - PORTED 2026-08-17 by row LTX25-GUIDED-VIDEO (#1092): the CROSS-attention +// guidance perturbations, `SKIP_A2V_CROSS_ATTN` and `SKIP_V2A_CROSS_ATTN` +// (guidance/perturbations.py:8-16, transformer.py:335,367 +// `cross_attn_skip_all`). They are `Ltx2DitPerturbation`'s two booleans. // -// THE SELF-ATTENTION HALF IS NOW PORTED, and this entry said the whole -// mechanism was unported until #1005. `Ltx2DitPerturbation` is upstream's -// `perturbations` argument (model.py:492) at the one batch size this port -// serves, and `Ltx2AttentionArgs::all_perturbed` is +// WHY THE PREVIOUS ENTRY WAS WRONG RATHER THAN MERELY STALE. It refused them +// on the ground that "nothing upstream that this port serves constructs +// them — STG is built from `stg_blocks` and reaches the SELF-attention types +// alone". STG does. The isolated-modality pass does not: `_guided_denoise` +// builds BOTH cross types with `blocks=None` whenever either guider has +// `modality_scale != 1.0` (denoisers.py:121-137, guiders.py:283-285). Every +// VIDEO row of the params table sets it to 3.0 +// (utils/constants.py:54, :64), so this was upstream's default on the video +// path the whole time. The sentence was true of text-to-audio, which pins +// the field to 1.0 (t2a_one_stage.py:202), and it was written while +// text-to-audio was the only guided path here. +// +// THE SELF-ATTENTION HALF was ported by #1005. `Ltx2DitPerturbation` is +// upstream's `perturbations` argument (model.py:493) at the one batch size +// this port serves, and `Ltx2AttentionArgs::all_perturbed` is // `use_attention = not all_perturbed` (attention.py:557). `nullptr` remains -// upstream's `perturbations=None` path (model.py:509-511) and is what every -// caller but text-to-audio passes. +// upstream's `perturbations=None` path (model.py:509-511) and is what an +// unguided phase still passes. // // The BATCHED form (`BatchedPerturbationConfig`, perturbations.py:53-143, -// indexed [type, block, SAMPLE]) is still unported, and so is the partial -// blend it exists for (`out * mask + v * (1 - mask)`, attention.py:571-572). +// indexed [type, block, SAMPLE]) is ported in `ltx2_pipeline.h` and reached +// by `Ltx2GuidedDenoise`, which builds one config over the pass list and +// slices it per pass. What is still unported is batch > 1 and the partial +// blend it exists for (`out * mask + v * (1 - mask)`, attention.py:572-573). // Both are degenerate at `Ltx2ModalityInput::batch == 1`, which is the only // batch any path here runs. // - The caption projections (text_projection.py:31-38). LTX-2.5 is a 22B-form @@ -533,7 +542,7 @@ struct Ltx2DitOutputs { std::vector audio; // [batch, audio tokens, audio_out_channels] }; -// `perturbations` on LTXModel.forward (model.py:492), reduced to what this port +// `perturbations` on LTXModel.forward (model.py:493), reduced to what this port // can express. One entry per BLOCK; `true` means that block's self-attention is // replaced by its value projection, which is STG. // @@ -541,21 +550,39 @@ struct Ltx2DitOutputs { // indexed [type, block, SAMPLE]; this is indexed [block] alone, because // `Ltx2ModalityInput::batch` is 1 on every path here and the sample axis is a // degenerate one. `Ltx2BatchedPerturbationConfig` (ltx2_pipeline.h) is the -// batched form and stays ungated by any product caller — recorded as owed in -// .agents/specs/ltx25-t2a-one-stage.md rather than silently bypassed. +// batched form, and row LTX25-GUIDED-VIDEO (#1092) is what gave it a product +// caller: `Ltx2GuidedDenoise` builds one config over all four passes and slices +// it per pass, which is `denoisers.py:182-187` and is where the ONE-sample +// flattening below happens. // // EMPTY IS NOT "NOTHING PERTURBED BY COINCIDENCE": a vector of the wrong length // is REFUSED, so a config built for a different layer count cannot silently // perturb the first N blocks and leave the rest alone. // -// The two CROSS-attention perturbation types upstream defines -// (SKIP_A2V_CROSS_ATTN, SKIP_V2A_CROSS_ATTN, perturbations.py:8-16) have no -// field here and are not ported. Nothing upstream that this port serves -// constructs them: STG is built from `stg_blocks` and reaches the self-attention -// types only (utils/constants.py:49-68 through guiders.py:194-211). +// ALL FOUR upstream perturbation types are represented here. The two CROSS +// directions arrived with #1092 and are booleans rather than per-block vectors, +// because the one thing that builds them asks for ALL blocks +// (`Perturbation(type=..., blocks=None)`, denoisers.py:132-135) and upstream's +// own reader is the per-block scalar `cross_attn_skip_all` (transformer.py:335, +// :367) rather than a mask multiply. A per-block cross vector would be a surface +// with no constructor. +// +// WHAT THIS ENTRY USED TO SAY, kept because the sentence was load-bearing and +// wrong: "Nothing upstream that this port serves constructs them: STG is built +// from `stg_blocks` and reaches the self-attention types only". That was true +// while text-to-audio was the only guided path here — it pins +// `modality_scale = 1.0` (t2a_one_stage.py:202), which is exactly the value +// `do_isolated_modality_generation` reads as OFF. Every VIDEO row defaults it to +// 3.0 (utils/constants.py:54, :64), so the isolated-modality pass is upstream's +// DEFAULT there and these two types are on the reachable path. struct Ltx2DitPerturbation { std::vector video_self_attn; // [num_layers], empty = none std::vector audio_self_attn; // [num_layers], empty = none + // `cross_attn_skip_all` on the VIDEO args, i.e. SKIP_A2V_CROSS_ATTN: the + // audio-to-video direction, which WRITES the video stream. + bool video_cross_attn_skip_all = false; + // `cross_attn_skip_all` on the AUDIO args, i.e. SKIP_V2A_CROSS_ATTN. + bool audio_cross_attn_skip_all = false; }; // LTXModel.forward (model.py:492-538), plus the preprocessors it drives @@ -625,6 +652,27 @@ struct Ltx2BlockArgs { // STG for THIS block (attention.py:552-577). See `Ltx2DitPerturbation`. bool video_self_attn_perturbed = false; bool audio_self_attn_perturbed = false; + // `cross_attn_skip_all` (transformer_args.py:70, read at transformer.py:335 + // and :367). THE FLAG RIDES ON THE STREAM BEING WRITTEN, not on the stream + // being read: `video.cross_attn_skip_all` skips A2V, which writes the VIDEO + // stream from audio keys, and `audio.cross_attn_skip_all` skips V2A. + // + // THIS COMMENT USED TO SAY A TEST COULD NOT SEPARATE THEM, and it was the + // wrong conclusion from a true premise. The premise: on the SHIPPED path both + // directions are off together, because `_guided_denoise` builds the + // isolated-modality pass with BOTH (denoisers.py:125-138), so swapping the two + // flags renders identically there. The conclusion does not follow, because + // nothing obliges the separating test to use the shipped combination. + // `run_a2v` and `run_v2a` read the two streams' `enabled` flags + // asymmetrically (transformer.py:265-269), so a forward with one stream + // PRESENT but DISABLED runs exactly one cross direction and each flag becomes + // observable alone. That is what + // "ltx2 dit: each CROSS perturbation gates ITS OWN direction and no other" + // does; it is red against a build that applies only one direction (M12, M13) + // and against a build that swaps them (M15), all three of which were GREEN + // over the shipped-path case alone. + bool video_cross_attn_skip_all = false; + bool audio_cross_attn_skip_all = false; // Audio<->video cross-attention AdaLN inputs (transformer_args.py:388-411). const float* video_cross_scale_shift = nullptr; // [batch, video tokens, 4 * dim] const float* video_cross_gate = nullptr; // [batch, 1, dim] diff --git a/include/vllm/model_executor/models/ltx2_denoisers.h b/include/vllm/model_executor/models/ltx2_denoisers.h new file mode 100644 index 000000000..f9d52e971 --- /dev/null +++ b/include/vllm/model_executor/models/ltx2_denoisers.h @@ -0,0 +1,189 @@ +// LTX-2.5 denoisers — `ltx-pipelines/utils/denoisers.py` @ Lightricks/LTX-2 +// fd4ded7f, in its own translation unit because upstream has its own file. +// +// Row LTX25-GUIDED-VIDEO, issue +// https://github.com/mudler/vllm.cpp/issues/1092. Spec +// .agents/specs/ltx25-guided-video.md. +// +// ── WHAT THIS FILE IS FOR ────────────────────────────────────────────────── +// +// Upstream has three denoisers (`SimpleDenoiser`, `GuidedDenoiser`, +// `FactoryGuidedDenoiser`) and they share ONE function: `_guided_denoise` +// (denoisers.py:61-211). That function is what this file ports. It is the piece +// four unported pipelines are each blocked on — `a2vid_two_stage.py:230`, +// `ti2vid_two_stages.py:248`, `ti2vid_two_stages_hq.py:271`, +// `keyframe_interpolation.py:232` — and it is what a `pipeline_kind = one_stage` +// render here was missing entirely: `ti2vid_one_stage.py:221-226` builds a +// `FactoryGuidedDenoiser` and this port ran one unguided forward per step. +// +// ── THE FOUR THINGS THAT ARE EASY TO GET WRONG AND STILL RENDER ──────────── +// +// 1. THE SPACE. The transformer this seam is handed is upstream's `X0Model` +// (built at blocks.py:480-482, forward at model.py:590-604), so every pass it +// returns is ALREADY `latent - sigma * velocity` and the guider combines +// denoised tensors. Combining velocities and converting once afterwards is a +// DIFFERENT function whenever `rescale_scale != 0` (guiders.py:268-271), and +// it is 0.7 on every video row of the params table. That defect shipped on +// the audio arm of this tree and is #1039. The conversion therefore lives in +// the caller's `Ltx2X0Model`, which is where upstream puts it, and the seam +// combines `Ltx2X0Outputs::video` / `::audio`. +// +// THAT IS CALLER DISCIPLINE AND NOT A TYPE GUARANTEE, and this comment +// claimed the stronger thing until 2026-08-17. `Ltx2X0Outputs` carries the +// raw velocity beside the denoised prediction (below), so a lambda that fills +// `video` with what belongs in `video_velocity` type-checks and renders. +// Nothing in the signature can stop it; the four per-arm invariants in +// `test_ltx2_video` do, and mutations M1-M4 — one per arm, each handing the +// seam a velocity — are red against them. A structural claim a type does not +// enforce is worth less than a gate that catches the substitution, so the +// gate is where this is argued. +// +// 2. ONE PASS LIST, TWO GUIDERS. `_guided_denoise` takes the UNION of what the +// two guiders want — one `uncond` pass if either asks (`:102-109`), one `ptb` +// pass carrying both modalities' `stg_blocks` (`:111-119`), one `mod` pass if +// either asks (`:121-137`) — and then combines each modality with its OWN +// guider over the same splits (`:203-204`). Running a per-modality pass list +// instead would issue up to six forwards where upstream issues four, and +// would hand the audio stream a different video state to cross-attend to on +// the video-only passes. Both renders finish. +// +// 3. `post_process_latent` IS NOT PART OF THE DENOISER. It is applied by the +// LOOP, to the guider's OUTPUT (`samplers.py:35`, `:484`), not to each arm on +// the way out of the forward. Pinning the conditioned tokens per arm makes +// every arm agree on those tokens, which silently zeroes the guidance delta +// exactly where a keyframe or a reference clip is conditioning. This seam +// therefore returns the raw guided prediction and the caller post-processes. +// +// 4. A SKIPPED STEP RUNS NO FORWARD. When both guiders skip, upstream returns +// the PREVIOUS step's denoised pair (`:87-90`) rather than running the +// conditional pass and using it. `skip_step` is 0 in every params table, so +// this is reachable only through an explicit request; it is ported because +// the request surface exposes it. +// +// ── WHAT IS NOT HERE ─────────────────────────────────────────────────────── +// +// The BATCHING. Upstream concatenates the passes along the batch axis and issues +// ONE transformer call (`:141-186`); this issues one call per pass. That is a +// throughput difference and not a numerical one at `batch == 1`, and it is the +// same adaptation `Ltx2DitPerturbation` already records: every path in this port +// runs `Ltx2ModalityInput::batch == 1`. + +#pragma once + +#include +#include +#include + +#include "vllm/model_executor/models/ltx2.h" +#include "vllm/model_executor/models/ltx2_pipeline.h" + +namespace vllm { + +// What one pass of the `X0Model` returns. `Ltx2DitOutputs` carries the raw +// velocity; this carries the denoised prediction AND the velocity it came from, +// because "which space was this combined in" is an arithmetic question between +// three tensors and cannot be answered from the denoised one alone. The velocity +// is what the trace records per arm and what the gate's per-arm invariant +// `x0 == latent - sigma * velocity` is checked against. +struct Ltx2X0Outputs { + std::vector video; // `to_denoised(video.latent, vx, video.timesteps)` + std::vector audio; // `to_denoised(audio.latent, ax, audio.timesteps)` + std::vector video_velocity; // the DiT's own output, before the conversion + std::vector audio_velocity; +}; + +// `X0Model` (model.py:590-604), supplied by the caller. +// +// It is a callable and not a (params, weights) pair on purpose. Upstream's +// `_guided_denoise(transformer, ...)` takes the model the same way, the host and +// the device forward are two lambdas over one seam, and the four pipelines that +// will use this next supply their own conditioning without this file learning +// anything about keyframes, reference clips or two-stage schedules. +// +// EITHER STREAM MAY BE NULL, which is upstream's absent modality. `perturbations` +// is null on the passes that have none, which is upstream's +// `PerturbationConfig.empty()` reaching `model.py:509-511`. +using Ltx2X0Model = std::function; + +// The pass names of `_guided_denoise` (`:100-137`), in the order it appends +// them. The order is not cosmetic: it is the batch order upstream splits back +// out at `:188-190`, and it is the order the perturbation config is built in. +enum class Ltx2DenoisePass { + kCond = 0, + kUncond = 1, + kPerturbed = 2, + kModality = 3, +}; +inline constexpr int64_t kLtx2DenoisePassCount = 4; + +struct Ltx2GuidedDenoiseInputs { + // The two streams, exactly as they would be handed to the forward for the + // conditional pass. The seam copies them per pass and overrides `context` and + // `enabled`; it never touches the latent, the timesteps or the positions. + const Ltx2ModalityInput* video = nullptr; + const Ltx2ModalityInput* audio = nullptr; + + // `guider.negative_context` (guiders.py:236). A null is upstream's + // `negative_context is None`, whose branch at `:107-108` falls back to the + // POSITIVE context rather than refusing — and which `:104-106` refuses when + // that modality's guider is the one asking. Both are mirrored. + const float* video_negative_context = nullptr; + const float* audio_negative_context = nullptr; + + Ltx2MultiModalGuiderParams video_guider; + Ltx2MultiModalGuiderParams audio_guider; + + // `transformer.num_blocks` (denoisers.py:180). Needed to size the perturbation + // masks, and refused when it disagrees with the DiT the caller's lambda drives + // — a mask built for another block count perturbs a prefix and renders. + int64_t num_blocks = 0; + + int64_t step_index = 0; + // `force_uncond_pass` (`:76`, read at `:102-103`). `RetakePipeline` is the one + // upstream caller that sets it (retake.py:305-311). + bool force_uncond_pass = false; + + // `_last_denoised_video` / `_last_denoised_audio` (denoisers.py:274-275). Null + // on the first step. A skipped step with nothing to reuse is refused rather + // than reading an empty vector. + const std::vector* last_denoised_video = nullptr; + const std::vector* last_denoised_audio = nullptr; +}; + +// Everything one call produced. The per-pass tensors are kept because a gate +// that can only see the combination cannot tell which arm was converted in which +// space, and because #1039's first gate covered one arm out of three and three +// mutations survived it. +struct Ltx2GuidedDenoiseResult { + // `DenoisedLatentResult.denoised` (utils/types.py), per modality. Empty when + // that modality was absent. + std::vector video_denoised; + std::vector audio_denoised; + bool video_skipped = false; + bool audio_skipped = false; + + // Which passes ran, and what each returned. Indexed by `Ltx2DenoisePass`. + bool pass_ran[kLtx2DenoisePassCount] = {false, false, false, false}; + std::vector video_pass[kLtx2DenoisePassCount]; // x0 + std::vector audio_pass[kLtx2DenoisePassCount]; + std::vector video_pass_velocity[kLtx2DenoisePassCount]; // raw + std::vector audio_pass_velocity[kLtx2DenoisePassCount]; + + // The blocks the perturbed pass actually asked the DiT to skip, read off the + // mask that was handed over rather than copied from the guider params. A + // config that is BUILT and not HANDED OVER is invisible in the params. + std::vector perturbed_video_blocks; + std::vector perturbed_audio_blocks; + // Whether the isolated-modality pass reached the DiT with BOTH cross + // directions off, observed at the call for the same reason. + bool modality_pass_skipped_a2v = false; + bool modality_pass_skipped_v2a = false; +}; + +// `_guided_denoise` (denoisers.py:61-211). +Ltx2GuidedDenoiseResult Ltx2GuidedDenoise(const Ltx2X0Model& transformer, + const Ltx2GuidedDenoiseInputs& in); + +} // namespace vllm diff --git a/include/vllm/multimodal/ltx2_video.h b/include/vllm/multimodal/ltx2_video.h index fe3007e6c..2ae214989 100644 --- a/include/vllm/multimodal/ltx2_video.h +++ b/include/vllm/multimodal/ltx2_video.h @@ -459,6 +459,53 @@ inline constexpr char kLtx2AudioRescaleScaleExtra[] = "audio_rescale_scale"; inline constexpr char kLtx2AudioSkipStepExtra[] = "audio_skip_step"; inline constexpr char kLtx2AudioStgBlocksExtra[] = "audio_stg_blocks"; +// THE VIDEO GUIDER, row LTX25-GUIDED-VIDEO (#1092). The same row of flags on the +// other stream, from the same parser (`default_1_stage_arg_parser`, +// utils/args.py:947-1066). ABSENT MEANS the params table's own value: 3.0 / 1.0 / +// 0.7 / 3.0 and block 28 on the 2.3-and-later lineage +// (utils/constants.py:40-88). +// +// THE MODALITY KNOBS EXIST HERE AND NOT ON THE T2A ROW ABOVE, and the asymmetry +// is upstream's rather than an oversight on either side. Text-to-audio has no +// video stream, so `t2a_one_stage.py:200-202` pins `modality_scale = 1.0` and +// exposes no flag. A joint render has both streams and the parser exposes +// `--a2v-guidance-scale` and `--v2a-guidance-scale`, which are the video and +// audio guiders' `modality_scale` respectively (utils/args.py:987-996 and its +// audio counterpart). Reaching either turns on a fourth DiT forward per step. +// +// EVERY ONE OF THESE IS REFUSED on a phase whose recipe sets +// `allow_guidance_override = false` — the distilled two-stage and retake +// recipes, whose guidance is distilled INTO the weights. Honouring an override +// there would sample a trajectory the weights were never trained for, which is +// the same argument `fixed_num_inference_steps` already makes for the schedule. +inline constexpr char kLtx2VideoCfgScaleExtra[] = "video_cfg_guidance_scale"; +inline constexpr char kLtx2VideoStgScaleExtra[] = "video_stg_guidance_scale"; +inline constexpr char kLtx2VideoRescaleScaleExtra[] = "video_rescale_scale"; +inline constexpr char kLtx2VideoSkipStepExtra[] = "video_skip_step"; +inline constexpr char kLtx2VideoStgBlocksExtra[] = "video_stg_blocks"; +inline constexpr char kLtx2A2vGuidanceScaleExtra[] = "a2v_guidance_scale"; +inline constexpr char kLtx2V2aGuidanceScaleExtra[] = "v2a_guidance_scale"; + +// THE NEGATIVE CONDITIONING FOR AN ENGINE WITH NO TEXT TOWER, and a LOCAL +// ADAPTATION recorded as one. +// +// Upstream has no embeds surface at all: every pipeline encodes +// `[prompt, negative_prompt]` in ONE `PromptEncoder` call +// (ti2vid_one_stage.py:166-174) and takes `.video_encoding` / `.audio_encoding` +// from each half. `prompt_embeds_path` and the `audio_prompt_embeds_path` extra +// are this port's own affordance for running the DiT without a 12B tower; these +// two are the SAME affordance applied to the second of upstream's two +// encodings, not a new concept. +// +// They are supplied together with each other, and only alongside the positive +// pair. Without them and without a tower, a guider that asks for the +// unconditional pass is REFUSED BY NAME rather than served the positive context +// twice — which would make `(cfg_scale - 1) * (cond - uncond)` identically zero +// and produce an unguided render wearing a guided render's configuration. +inline constexpr char kLtx2NegativePromptEmbedsExtra[] = "negative_prompt_embeds_path"; +inline constexpr char kLtx2NegativeAudioPromptEmbedsExtra[] = + "negative_audio_prompt_embeds_path"; + // WHAT THE LAST `Generate()` ACTUALLY HANDED THE DiT's CROSS-ATTENTION. // // Every field is read off the exact f32 buffers `Ltx2ModalityInput::context` @@ -773,6 +820,65 @@ struct Ltx2ConditioningTrace { std::vector t2a_first_next_latent; double t2a_first_sigma = 0.0; + // ── the GUIDED VIDEO denoise, row LTX25-GUIDED-VIDEO (#1092) ────────────── + // + // Everything the FIRST step of the FIRST phase produced, and nothing else. One + // step decides every question below, and recording every step would hold a + // whole real trajectory in memory. + // + // WHY THE VELOCITIES SIT BESIDE THE X0 TENSORS. "Which space was this combined + // in" is an equation between three tensors — `x0 == latent - sigma*velocity` — + // and cannot be answered from the x0 tensor alone. It is exact in x0 space and + // off by the whole sample in velocity space, which is what makes it a gate + // rather than a tolerance. Recorded PER ARM, because a claim about "every + // pass" made from one recorded pass is a claim about a quarter of them: + // #1039's first gate covered the conditional arm alone and three mutations + // survived it. + bool video_guided = false; + int64_t video_cond_forwards = 0; + int64_t video_uncond_forwards = 0; + int64_t video_perturbed_forwards = 0; + int64_t video_modality_forwards = 0; + // Read off the mask handed to the DiT, not copied from the guider params: a + // perturbation that is BUILT and not HANDED OVER leaves the params untouched + // and the render finite. + std::vector video_perturbed_blocks; + std::vector video_audio_perturbed_blocks; + bool video_modality_skipped_a2v = false; + bool video_modality_skipped_v2a = false; + // The guidance phase 0 resolved, after the request overrides. The gate replays + // `Ltx2MultiModalGuidance` over the recorded arms with these, so a build that + // resolved different params fails the replay instead of agreeing with itself. + double video_guidance_cfg_scale = 0.0; + double video_guidance_stg_scale = 0.0; + double video_guidance_rescale_scale = 0.0; + double video_guidance_modality_scale = 0.0; + std::vector video_first_latent; + std::vector video_first_cond_velocity; + std::vector video_first_cond; + std::vector video_first_uncond_velocity; + std::vector video_first_uncond; + std::vector video_first_perturbed_velocity; + std::vector video_first_perturbed; + std::vector video_first_modality_velocity; + std::vector video_first_modality; + // The guider's output BEFORE `post_process_latent`, which is what + // `Ltx2MultiModalGuidance` returned and what the replay must reproduce + // exactly. + std::vector video_first_denoised; + // And AFTER it, which is what the stepper was handed. Two fields rather than + // one, because `post_process_latent` is the identity whenever no token is + // conditioned and a single field could not say which of the two a build passed + // on. + std::vector video_first_stepper_input; + std::vector video_first_next_latent; + double video_first_sigma = 0.0; + // The PER-TOKEN timesteps step 0 ran at, so the invariant is checked with the + // same sigma `ToDenoised` used rather than with the schedule scalar — they + // differ exactly where a token is conditioned, which is where getting it wrong + // re-noises a keyframe. + std::vector video_first_timesteps; + // True only once the `Generate` that produced this conditioning RETURNED. The // trace is filled immediately after the connector and BEFORE the denoise loop, // because that is the only point at which the exact buffers cross-attention diff --git a/src/vllm/model_executor/models/ltx2_denoisers.cpp b/src/vllm/model_executor/models/ltx2_denoisers.cpp new file mode 100644 index 000000000..5c12acfd3 --- /dev/null +++ b/src/vllm/model_executor/models/ltx2_denoisers.cpp @@ -0,0 +1,368 @@ +// `_guided_denoise` (ltx-pipelines utils/denoisers.py:61-211) at +// Lightricks/LTX-2 @ fd4ded7f. See ltx2_denoisers.h for the four things that +// fail silently if guessed. +// +// Row LTX25-GUIDED-VIDEO, issue #1092. + +#include "vllm/model_executor/models/ltx2_denoisers.h" + +#include +#include +#include +#include + +namespace vllm { +namespace { + +[[noreturn]] void Fail(const std::string& why) { + throw std::runtime_error("ltx2 guided denoise: " + why); +} + +// `_POSITIVE_ONLY_GUIDER` (denoisers.py:25-28) is +// `MultiModalGuiderParams(cfg_scale=1.0, stg_scale=0.0, modality_scale=1.0)`, +// which is `Ltx2MultiModalGuiderParams`'s own default construction. Stated as a +// function rather than inlined so the identity is checkable by eye against +// `_ensure_guider` (`:31-33`). +Ltx2MultiModalGuiderParams PositiveOnlyGuider() { return Ltx2MultiModalGuiderParams{}; } + +// `perturbations.mask(type, block)` collapsed to this port's one sample. The +// KEEP polarity is upstream's — 1 keeps, 0 perturbs (perturbations.py:53-56) — +// so the DiT flag, which is `all_perturbed`, is the negation. +bool PerturbedAt(const Ltx2BatchedPerturbationConfig& config, Ltx2PerturbationType type, + int64_t block, int64_t sample) { + const std::vector mask = config.Mask(type, block); + return mask[static_cast(sample)] == 0; +} + +} // namespace + +Ltx2GuidedDenoiseResult Ltx2GuidedDenoise(const Ltx2X0Model& transformer, + const Ltx2GuidedDenoiseInputs& in) { + if (in.video == nullptr && in.audio == nullptr) { + // `transformer.py:259-260` refuses it upstream, one level lower. + Fail("both modalities are null; at least one of `video` or `audio` must be provided"); + } + if (in.num_blocks < 1) { + Fail("`num_blocks` is " + std::to_string(in.num_blocks) + + "; the perturbation masks are sized by it (denoisers.py:180) and a wrong count " + "perturbs a prefix of the blocks and renders"); + } + + // `_ensure_guider` (denoisers.py:31-33): an ABSENT modality takes the + // positive-only guider, so its `calculate` returns `cond` unchanged and it asks + // for no extra pass. A modality that is PRESENT keeps its caller's guider even + // when every scale is at its no-op value. + const Ltx2MultiModalGuiderParams video_guider = + in.video != nullptr ? in.video_guider : PositiveOnlyGuider(); + const Ltx2MultiModalGuiderParams audio_guider = + in.audio != nullptr ? in.audio_guider : PositiveOnlyGuider(); + + Ltx2GuidedDenoiseResult result; + + // `should_skip_step` (denoisers.py:84-85). + const bool v_skip = video_guider.ShouldSkipStep(in.step_index); + const bool a_skip = audio_guider.ShouldSkipStep(in.step_index); + result.video_skipped = v_skip; + result.audio_skipped = a_skip; + + // `if v_skip and a_skip` (`:87-90`) — NO FORWARD AT ALL. Running the + // conditional pass and using it is the plausible reading of "skip the + // guidance" and is a whole DiT forward per skipped step and a different + // trajectory, on a render that finishes either way. + auto reuse = [&](const std::vector* last, const char* which) { + if (last == nullptr || last->empty()) { + Fail(std::string("step ") + std::to_string(in.step_index) + + " skips the " + which + + " guider and no earlier step produced a denoised prediction to reuse. " + "`should_skip_step` is `step % (skip_step + 1) != 0` (guiders.py:287-291), which is " + "false at step 0, so this is unreachable through the request surface and is a defect " + "rather than a bad request"); + } + return *last; + }; + if (v_skip && a_skip) { + if (in.video != nullptr) result.video_denoised = reuse(in.last_denoised_video, "video"); + if (in.audio != nullptr) result.audio_denoised = reuse(in.last_denoised_audio, "audio"); + return result; + } + + // ── the pass list (denoisers.py:97-137) ─────────────────────────────────── + // + // ONE list for BOTH modalities, and the union of what the two guiders want. + // See ltx2_denoisers.h item 2 for the render a per-modality list produces. + struct Pass { + Ltx2DenoisePass kind; + const float* video_context; + const float* audio_context; + Ltx2PerturbationConfig perturbation; + }; + std::vector passes; + + const float* v_context = in.video != nullptr ? in.video->context : nullptr; + const float* a_context = in.audio != nullptr ? in.audio->context : nullptr; + if (in.video != nullptr && v_context == nullptr) { + Fail("v_context is required when video_state is provided (denoisers.py:92-93)"); + } + if (in.audio != nullptr && a_context == nullptr) { + Fail("a_context is required when audio_state is provided (denoisers.py:94-95)"); + } + passes.push_back({Ltx2DenoisePass::kCond, v_context, a_context, Ltx2PerturbationConfig{}}); + + // `:102-109`. `force_uncond_pass` adds the pass for a modality that is PRESENT + // even when its own guider does not ask (retake.py:305-311 is the one upstream + // caller that sets it). + const bool v_needs_neg = video_guider.DoUnconditionalGeneration() || + (in.force_uncond_pass && in.video != nullptr); + const bool a_needs_neg = audio_guider.DoUnconditionalGeneration() || + (in.force_uncond_pass && in.audio != nullptr); + if (v_needs_neg || a_needs_neg) { + if (v_needs_neg && in.video_negative_context == nullptr) { + Fail("negative context is required for unconditioned denoising on the VIDEO stream " + "(denoisers.py:104-105). `do_unconditional_generation` is " + "`not isclose(cfg_scale, 1.0)` (guiders.py:275-277), so either supply the negative " + "conditioning or set the video cfg scale to 1.0"); + } + if (a_needs_neg && in.audio_negative_context == nullptr) { + Fail("negative context is required for unconditioned denoising on the AUDIO stream " + "(denoisers.py:106-107). `do_unconditional_generation` is " + "`not isclose(cfg_scale, 1.0)` (guiders.py:275-277), so either supply the negative " + "conditioning or set the audio cfg scale to 1.0"); + } + // `:108-109` — a modality with no negative context falls back to its POSITIVE + // one rather than being dropped from the pass. That is not a defensive + // default: it is how a pass forced for the OTHER modality still carries a + // legal context for this one. + passes.push_back({Ltx2DenoisePass::kUncond, + in.video_negative_context != nullptr ? in.video_negative_context : v_context, + in.audio_negative_context != nullptr ? in.audio_negative_context : a_context, + Ltx2PerturbationConfig{}}); + } + + // `:111-119`. ONE perturbed pass carrying BOTH modalities' blocks. + { + Ltx2PerturbationConfig stg; + if (video_guider.DoPerturbedGeneration()) { + Ltx2Perturbation p; + p.type = Ltx2PerturbationType::kSkipVideoSelfAttn; + p.blocks = video_guider.stg_blocks; + stg.perturbations.push_back(std::move(p)); + } + if (audio_guider.DoPerturbedGeneration()) { + Ltx2Perturbation p; + p.type = Ltx2PerturbationType::kSkipAudioSelfAttn; + p.blocks = audio_guider.stg_blocks; + stg.perturbations.push_back(std::move(p)); + } + // A BLOCK LIST THAT MISSES EVERY BLOCK IS A WASTED FORWARD AND A ZERO TERM. + // `Perturbation.is_perturbed` is `block in self.blocks` + // (perturbations.py:26-33), so `stg_blocks = [28]` on a model with fewer + // blocks perturbs nothing: the perturbed pass returns the conditional pass's + // own tensor and `stg_scale * (cond - perturbed)` is exactly zero. The render + // is finite, the right size, and carries no spatio-temporal guidance at all. + // Upstream never meets this because it only ever runs 48-block checkpoints; + // this port runs reduced ones, and a smaller checkpoint is a legal thing to + // hand it. + // + // AN EMPTY LIST IS EXEMPT, and it was not until 2026-08-17. `blocks=[]` is + // upstream's documented spelling for "perturb no block", distinct from + // `blocks=None`'s "perturb every block" (perturbations.py:26-33), named as + // the way to disable STG at `ltx-pipelines/docs/multimodal-guidance.md:13`, + // shipped in `LTX_2_3_HQ_PARAMS` (constants.py:105, :113), and reachable + // through `nargs="*"` (args.py:979-985). Upstream runs the pass and takes + // the zero term; so does this. What is refused is a list that NAMES blocks + // and reaches none of them, which is a request that disagrees with the + // CHECKPOINT rather than a caller who asked for nothing. + const auto check_reaches_a_block = [&](const Ltx2MultiModalGuiderParams& guider, + const char* which) { + if (!guider.DoPerturbedGeneration()) return; + if (guider.stg_blocks.empty()) return; + for (const int64_t block : guider.stg_blocks) { + if (block >= 0 && block < in.num_blocks) return; + } + Fail(std::string("the ") + which + " STG scale is " + std::to_string(guider.stg_scale) + + " and none of its " + std::to_string(guider.stg_blocks.size()) + + " stg_blocks is in range for this DiT's " + std::to_string(in.num_blocks) + + " blocks, so the perturbed forward would be identical to the conditional one and " + "`stg_scale * (cond - perturbed)` would be exactly zero (guiders.py:264). Name blocks " + "this checkpoint has, or set the STG scale to 0.0"); + }; + if (!stg.perturbations.empty()) { + check_reaches_a_block(video_guider, "video"); + check_reaches_a_block(audio_guider, "audio"); + passes.push_back({Ltx2DenoisePass::kPerturbed, v_context, a_context, std::move(stg)}); + } + } + + // `:121-137`. The isolated-modality pass: BOTH cross directions, ALL blocks + // (`blocks=None`), when EITHER guider isolates. `modality_scale` is 3.0 on + // every video row of the params table (utils/constants.py:54, :64), so this is + // the default arm rather than a corner. + if (video_guider.DoIsolatedModalityGeneration() || + audio_guider.DoIsolatedModalityGeneration()) { + Ltx2PerturbationConfig mod; + Ltx2Perturbation a2v; + a2v.type = Ltx2PerturbationType::kSkipA2vCrossAttn; + a2v.all_blocks = true; + Ltx2Perturbation v2a; + v2a.type = Ltx2PerturbationType::kSkipV2aCrossAttn; + v2a.all_blocks = true; + mod.perturbations.push_back(std::move(a2v)); + mod.perturbations.push_back(std::move(v2a)); + passes.push_back({Ltx2DenoisePass::kModality, v_context, a_context, std::move(mod)}); + } + + // ── the perturbation config (denoisers.py:182-187) ─────────────────────── + // + // ONE batched config over the whole pass list, then one sample slice per pass, + // which is upstream's `batched_ptb_configs` followed by the per-sample mask the + // block reads. Building a fresh single-sample config per pass would be + // arithmetically identical and would leave `Ltx2BatchedPerturbationConfig` — + // the shared seam that mirrors `BatchedPerturbationConfig` — with no product + // caller, which is the defect #1049 records. + const int64_t pass_count = static_cast(passes.size()); + std::vector configs; + configs.reserve(passes.size()); + for (const Pass& p : passes) configs.push_back(p.perturbation); + const Ltx2BatchedPerturbationConfig batched(configs, in.num_blocks); + + // ── run the passes (`:186`, one call there, `pass_count` calls here) ─────── + for (int64_t index = 0; index < pass_count; ++index) { + const Pass& pass = passes[static_cast(index)]; + + const Ltx2BatchedPerturbationConfig slice = batched.BatchSlice(index, index + 1); + Ltx2DitPerturbation perturbation; + bool any = false; + for (int64_t block = 0; block < in.num_blocks; ++block) { + const bool v = + PerturbedAt(slice, Ltx2PerturbationType::kSkipVideoSelfAttn, block, /*sample=*/0); + const bool a = + PerturbedAt(slice, Ltx2PerturbationType::kSkipAudioSelfAttn, block, /*sample=*/0); + if (v || a) any = true; + // Both vectors are sized whenever either is, because `Ltx2DitForward` + // refuses a vector that is neither empty nor one entry per block and an + // empty one means "nothing perturbed" for that stream. + perturbation.video_self_attn.push_back(v ? 1 : 0); + perturbation.audio_self_attn.push_back(a ? 1 : 0); + } + // The cross flags are not per block, because `Ltx2DitPerturbation` has no + // per-block cross vector and upstream's reader is the per-block scalar + // `cross_attn_skip_all` (transformer.py:335,367) rather than a mask + // multiply. That flattening is only sound while the config says the same + // thing on every block, which is what `blocks=None` produces + // (denoisers.py:132-135) — so it is CHECKED here rather than assumed. A + // block-list cross perturbation would otherwise be silently widened to all + // blocks, which renders. + const auto flatten_cross = [&](Ltx2PerturbationType type, const char* name) { + const bool first = PerturbedAt(slice, type, /*block=*/0, /*sample=*/0); + for (int64_t block = 1; block < in.num_blocks; ++block) { + if (PerturbedAt(slice, type, block, /*sample=*/0) == first) continue; + Fail(std::string("the ") + name + + " cross-attention perturbation differs between block 0 and block " + + std::to_string(block) + ". `Ltx2DitPerturbation` carries one boolean per direction " + "because the only thing upstream builds these with is `blocks=None` " + "(denoisers.py:132-135); a per-block cross perturbation cannot be represented and " + "would be widened to every block rather than refused"); + } + return first; + }; + perturbation.video_cross_attn_skip_all = + flatten_cross(Ltx2PerturbationType::kSkipA2vCrossAttn, "audio-to-video"); + perturbation.audio_cross_attn_skip_all = + flatten_cross(Ltx2PerturbationType::kSkipV2aCrossAttn, "video-to-audio"); + if (perturbation.video_cross_attn_skip_all || perturbation.audio_cross_attn_skip_all) { + any = true; + } + if (!any) { + // `PerturbationConfig.empty()` reaches the forward as upstream's + // `perturbations=None` (model.py:509-511), not as an all-ones mask, so the + // conditional and unconditional passes take the same path an unguided + // render takes. + perturbation = Ltx2DitPerturbation{}; + } + + Ltx2ModalityInput video_in; + Ltx2ModalityInput audio_in; + if (in.video != nullptr) { + video_in = *in.video; + video_in.context = pass.video_context; + // `enabled=not v_skip` (`:158`). A skipped modality stays PRESENT, so the + // other stream's cross attention still reads its latent + // (transformer.py:269 tests presence, not `enabled`). + video_in.enabled = !v_skip; + } + if (in.audio != nullptr) { + audio_in = *in.audio; + audio_in.context = pass.audio_context; + audio_in.enabled = !a_skip; + } + + const bool perturbed = any; + Ltx2X0Outputs out = transformer(in.video != nullptr ? &video_in : nullptr, + in.audio != nullptr ? &audio_in : nullptr, + perturbed ? &perturbation : nullptr); + + const size_t slot = static_cast(pass.kind); + result.pass_ran[slot] = true; + result.video_pass[slot] = std::move(out.video); + result.audio_pass[slot] = std::move(out.audio); + result.video_pass_velocity[slot] = std::move(out.video_velocity); + result.audio_pass_velocity[slot] = std::move(out.audio_velocity); + + // Observed at the call rather than restated from the guider params: a + // perturbation that is BUILT and not HANDED OVER leaves the params untouched + // and the render finite. Derived here, so a mutation that drops the argument + // moves this record. + if (pass.kind == Ltx2DenoisePass::kPerturbed && perturbed) { + for (int64_t block = 0; block < in.num_blocks; ++block) { + if (perturbation.video_self_attn[static_cast(block)] != 0) { + result.perturbed_video_blocks.push_back(block); + } + if (perturbation.audio_self_attn[static_cast(block)] != 0) { + result.perturbed_audio_blocks.push_back(block); + } + } + } + if (pass.kind == Ltx2DenoisePass::kModality && perturbed) { + result.modality_pass_skipped_a2v = perturbation.video_cross_attn_skip_all; + result.modality_pass_skipped_v2a = perturbation.audio_cross_attn_skip_all; + } + } + + // ── the combination (`:192-204`) ────────────────────────────────────────── + // + // EACH MODALITY WITH ITS OWN GUIDER, over the SAME splits. `r.get("uncond", + // (0.0, 0.0))` is upstream's absent pass and is the float 0.0 its + // `calculate` signature admits; a null here is the same thing, and + // `Ltx2MultiModalGuidance` reads it as 0.0 (guiders.py:247-249). + const auto at = [&result](const std::vector* passes_array, Ltx2DenoisePass kind) { + const size_t slot = static_cast(kind); + return result.pass_ran[slot] ? passes_array[slot].data() : nullptr; + }; + const size_t cond_slot = static_cast(Ltx2DenoisePass::kCond); + + if (in.video != nullptr) { + if (v_skip) { + result.video_denoised = reuse(in.last_denoised_video, "video"); + } else { + const std::vector& cond = result.video_pass[cond_slot]; + result.video_denoised = Ltx2MultiModalGuidance( + video_guider, cond.data(), at(result.video_pass, Ltx2DenoisePass::kUncond), + at(result.video_pass, Ltx2DenoisePass::kPerturbed), + at(result.video_pass, Ltx2DenoisePass::kModality), static_cast(cond.size())); + } + } + if (in.audio != nullptr) { + if (a_skip) { + result.audio_denoised = reuse(in.last_denoised_audio, "audio"); + } else { + const std::vector& cond = result.audio_pass[cond_slot]; + result.audio_denoised = Ltx2MultiModalGuidance( + audio_guider, cond.data(), at(result.audio_pass, Ltx2DenoisePass::kUncond), + at(result.audio_pass, Ltx2DenoisePass::kPerturbed), + at(result.audio_pass, Ltx2DenoisePass::kModality), static_cast(cond.size())); + } + } + return result; +} + +} // namespace vllm diff --git a/src/vllm/model_executor/models/ltx2_dit.cpp b/src/vllm/model_executor/models/ltx2_dit.cpp index bdf6b98cf..0dd7583eb 100644 --- a/src/vllm/model_executor/models/ltx2_dit.cpp +++ b/src/vllm/model_executor/models/ltx2_dit.cpp @@ -349,7 +349,14 @@ void Ltx2TransformerBlockForward(vt::Device device, const Ltx2DitParams& params, /*mod_index=*/0); }; - if (run_a2v) { + // `if run_a2v and not video.cross_attn_skip_all` (transformer.py:335). The + // guard is INSIDE the `run_a2v || run_v2a` block, exactly as upstream's is, + // so a pass that skips one direction still took the `vx_pre` / `ax_pre` + // snapshot above and the surviving direction reads the pre-cross state. + // Hoisting it into the outer condition would be equivalent only while both + // directions are always skipped together, which is true of the one caller + // today and is not a property of the flag. + if (run_a2v && !args.video_cross_attn_skip_all) { std::vector scale_v, shift_v, scale_a, shift_a; av_scale_shift(w.scale_shift_table_a2v_ca_video, args.video_cross_scale_shift, tv, dim, 0, &scale_v, &shift_v); @@ -377,7 +384,8 @@ void Ltx2TransformerBlockForward(vt::Device device, const Ltx2DitParams& params, AddGatedBroadcast(video_x, out, gate, batch, tv, dim); } - if (run_v2a) { + // `if run_v2a and not audio.cross_attn_skip_all` (transformer.py:367). + if (run_v2a && !args.audio_cross_attn_skip_all) { std::vector scale_a, shift_a, scale_v, shift_v; av_scale_shift(w.scale_shift_table_a2v_ca_audio, args.audio_cross_scale_shift, ta, adim, 2, &scale_a, &shift_a); @@ -822,7 +830,7 @@ Ltx2DitOutputs Ltx2DitForward(vt::Device device, const Ltx2DitParams& params, params.audio_num_attention_heads, have_both ? video : nullptr); } - // `perturbations` (model.py:492). A vector that is not exactly `num_layers` + // `perturbations` (model.py:493). A vector that is not exactly `num_layers` // long is REFUSED rather than indexed defensively: a config built for another // layer count would otherwise perturb a prefix of the blocks and leave the rest // alone, which is a legal-looking STG pass over the wrong blocks and renders. @@ -868,6 +876,11 @@ Ltx2DitOutputs Ltx2DitForward(vt::Device device, const Ltx2DitParams& params, perturbations->video_self_attn[static_cast(i)] != 0; a.audio_self_attn_perturbed = !perturbations->audio_self_attn.empty() && perturbations->audio_self_attn[static_cast(i)] != 0; + // Not indexed by block: the only thing that builds these asks for ALL + // blocks (`blocks=None`, denoisers.py:132-135), and upstream's reader is a + // per-block scalar rather than a mask multiply (transformer.py:335,367). + a.video_cross_attn_skip_all = perturbations->video_cross_attn_skip_all; + a.audio_cross_attn_skip_all = perturbations->audio_cross_attn_skip_all; } a.video_timestep_modulation = vs.modulation.empty() ? nullptr : vs.modulation.data(); a.audio_timestep_modulation = as.modulation.empty() ? nullptr : as.modulation.data(); diff --git a/src/vllm/multimodal/ltx2_video.cpp b/src/vllm/multimodal/ltx2_video.cpp index 6201dd748..2cd0be1af 100644 --- a/src/vllm/multimodal/ltx2_video.cpp +++ b/src/vllm/multimodal/ltx2_video.cpp @@ -30,6 +30,7 @@ #include "vllm/model_executor/models/ltx2_audio_vae.h" #include "vllm/model_executor/models/ltx2_conditioning.h" #include "vllm/model_executor/models/ltx2_connector.h" +#include "vllm/model_executor/models/ltx2_denoisers.h" #include "vllm/model_executor/models/ltx2_device.h" #include "vllm/model_executor/models/ltx2_dfr.h" #include "vllm/model_executor/models/ltx2_image_preprocess.h" @@ -373,13 +374,14 @@ constexpr char kLtx2DurationHeadPathExtra[] = "duration_head_path"; // they are no longer trusted: the list below is derived from this file on every // run and compared, and the failure prints the replacement to paste in. // READER ANCHORS (derived and gated by test_ltx2_video): -// 791 801 802 864 960 976 978 1069 1094 1199 1240 +// 798 808 809 871 967 983 985 1076 1101 1206 1247 1289 1291 const char* const kKnownLoadExtras[] = { kLtx2AudioPromptEmbedsExtra, kLtx2PipelineKindExtra, kLtx2ModelVersionExtra, kLtx2AllowUnportedExtra, kLtx2MaxPhaseExtra, kLtx2DitConfigPathExtra, kLtx2PromptValidRowsExtra, kLtx2EncoderConfigPathExtra, "upsampler_path", kLtx2DurationHeadPathExtra, kLtx2LoraPathExtra, kLtx2LoraStrengthExtra, + kLtx2NegativePromptEmbedsExtra, kLtx2NegativeAudioPromptEmbedsExtra, }; // FNV-1a over the raw bytes of a float buffer — the `Ltx2ConditioningTrace` @@ -624,6 +626,11 @@ struct Ltx2VideoEngine::Impl { // did on every checkpoint. std::vector video_prompt_embeds, audio_prompt_embeds; int64_t prompt_tokens = 0; + // The NEGATIVE half of the same fallback (row LTX25-GUIDED-VIDEO, #1092). + // Empty when the load supplied none, which is what makes a guider that asks + // for the unconditional forward a refusal rather than a silent reuse of the + // positive context. + std::vector negative_video_prompt_embeds, negative_audio_prompt_embeds; // The connector's CONFIGURATION is kept; its WEIGHTS are not. They are ~8 GB // of f32 at the shipped widths (ltx2_loader.h), the conditioning they process @@ -1269,6 +1276,60 @@ std::unique_ptr Ltx2VideoEngine::Load(const VideoModelParams& p im.video_prompt_embeds = encoded.video; im.audio_prompt_embeds = encoded.audio; } + + // ── the NEGATIVE half (row LTX25-GUIDED-VIDEO, #1092) ────────────────── + // + // The same two files for upstream's second encoding. Loaded here, INSIDE the + // positive block, because a negative pair without a positive one conditions + // nothing: `prompt_embeds_path` is what a tower-less engine renders from. + // The two negative files follow the positive pair's own rule — supplied + // together or not at all — and must agree with it on row count, because the + // guidance delta subtracts them elementwise. + const std::string negative_video_path = + VideoExtra(params.extras, kLtx2NegativePromptEmbedsExtra); + const std::string negative_audio_path = + VideoExtra(params.extras, kLtx2NegativeAudioPromptEmbedsExtra); + if (negative_video_path.empty() != negative_audio_path.empty()) { + Fail("the '" + std::string(kLtx2NegativePromptEmbedsExtra) + "' and '" + + std::string(kLtx2NegativeAudioPromptEmbedsExtra) + + "' extras are supplied together or not at all, for the same reason the positive pair " + "is: LTX-2.5 conditions TWO streams at two widths and one of them alone would leave a " + "stream unconditioned on the unconditional forward, which renders."); + } + if (!negative_video_path.empty()) { + im.negative_video_prompt_embeds = + ReadF32File(kLtx2NegativePromptEmbedsExtra, negative_video_path); + im.negative_audio_prompt_embeds = + ReadF32File(kLtx2NegativeAudioPromptEmbedsExtra, negative_audio_path); + if (static_cast(im.negative_video_prompt_embeds.size()) != v_rows * vw || + static_cast(im.negative_audio_prompt_embeds.size()) != a_rows * aw) { + Fail("the negative prompt embeds hold " + + std::to_string(im.negative_video_prompt_embeds.size()) + " / " + + std::to_string(im.negative_audio_prompt_embeds.size()) + + " floats and the positive pair holds " + + std::to_string(im.video_prompt_embeds.size()) + " / " + + std::to_string(im.audio_prompt_embeds.size()) + + " at widths " + std::to_string(vw) + " / " + std::to_string(aw) + + ". Upstream encodes `[prompt, negative_prompt]` in ONE call, so the two halves " + "share a padded width by construction and `(cfg_scale - 1) * (cond - uncond)` " + "subtracts them elementwise"); + } + if (im.has_connector) { + // Through the SAME connector, with the SAME mask. A negative stream that + // skipped it would be compared against a positive stream that did not, + // and the delta would be dominated by the connector rather than by the + // prompt. + std::vector additive(static_cast(v_rows), 0.0f); + for (int64_t s = im.prompt_valid_rows; s < v_rows; ++s) { + additive[static_cast(s)] = -std::numeric_limits::max(); + } + const Ltx2ConnectorEmbeddings encoded = RunConnector( + dit_file, im.video_connector_cfg, im.audio_connector_cfg, + im.negative_video_prompt_embeds, im.negative_audio_prompt_embeds, additive, v_rows); + im.negative_video_prompt_embeds = encoded.video; + im.negative_audio_prompt_embeds = encoded.audio; + } + } } return engine; } @@ -1433,6 +1494,157 @@ void AssertGeneratedKeyframesSupported(bool has_embedding, const std::string& di "admitted upstream (#902). Supply a generated-keyframe checkpoint, or drop the request."); } +// ── the guiders (row LTX25-GUIDED-VIDEO, #1092) ──────────────────────────── + +// `--*-stg-blocks`, `nargs="*"` (utils/args.py:979-985, :1039-1045). An extra +// that is PRESENT and empty is upstream's empty list — "perturb nothing" — and +// stays distinct from an ABSENT extra, which takes the params table's own value. +// Collapsing the two would make `video_stg_blocks=` silently mean block 28. +void ApplyStgBlocksExtra(const std::map& extras, const char* key, + std::vector* blocks) { + const auto at = extras.find(key); + if (at == extras.end()) return; + blocks->clear(); + const std::string& raw = at->second; + for (size_t i = 0; i < raw.size();) { + const size_t comma = raw.find(',', i); + const std::string token = raw.substr(i, comma == std::string::npos ? comma : comma - i); + if (!token.empty()) { + try { + blocks->push_back(std::stoll(token)); + } catch (const std::exception&) { + Fail("'" + std::string(key) + "' holds '" + token + + "', which is not an integer block index"); + } + } + if (comma == std::string::npos) break; + i = comma + 1; + } +} + +// One CLI flag each, from `default_1_stage_arg_parser` (utils/args.py:947-1066: +// the video row's six flags open at :948 and the audio row's at :1008). Each extra overrides ONE +// field of the phase's own resolved guider, which is what one flag does. +// +// REFUSED WHOLESALE on a phase that fixes its guidance. `allow_guidance_override +// = false` is set by the distilled two-stage and retake recipes +// (ltx2_recipes.py:125-158, retake.py:53) whose guidance is distilled INTO the +// weights, and until this row nothing read it. Honouring an override there would +// sample a trajectory the weights were never trained for — the same argument +// `fixed_num_inference_steps` already makes about the schedule, and the same +// reason it is a refusal rather than a silent clamp. +void ApplyGuidanceOverrides(const std::map& extras, + const Ltx2PhaseRecipe& phase, Ltx2MultiModalGuiderParams* video, + Ltx2MultiModalGuiderParams* audio) { + static const char* const kVideoKeys[] = { + kLtx2VideoCfgScaleExtra, kLtx2VideoStgScaleExtra, kLtx2VideoRescaleScaleExtra, + kLtx2VideoSkipStepExtra, kLtx2VideoStgBlocksExtra, kLtx2A2vGuidanceScaleExtra, + kLtx2AudioCfgScaleExtra, kLtx2AudioStgScaleExtra, kLtx2AudioRescaleScaleExtra, + kLtx2AudioSkipStepExtra, kLtx2AudioStgBlocksExtra, kLtx2V2aGuidanceScaleExtra}; + if (!phase.allow_guidance_override) { + for (const char* key : kVideoKeys) { + if (extras.find(key) == extras.end()) continue; + Fail("phase '" + phase.name + + "' fixes its own guidance, so the '" + std::string(key) + + "' extra is refused rather than applied. This recipe's scales are distilled INTO the " + "weights (ltx2_recipes.py:125-158), and a render that honoured the override would " + "sample a trajectory they were never trained for."); + } + return; + } + video->cfg_scale = ExtraDouble(extras, kLtx2VideoCfgScaleExtra, video->cfg_scale); + video->stg_scale = ExtraDouble(extras, kLtx2VideoStgScaleExtra, video->stg_scale); + video->rescale_scale = ExtraDouble(extras, kLtx2VideoRescaleScaleExtra, video->rescale_scale); + video->modality_scale = ExtraDouble(extras, kLtx2A2vGuidanceScaleExtra, video->modality_scale); + video->skip_step = ExtraInt(extras, kLtx2VideoSkipStepExtra, video->skip_step); + ApplyStgBlocksExtra(extras, kLtx2VideoStgBlocksExtra, &video->stg_blocks); + + audio->cfg_scale = ExtraDouble(extras, kLtx2AudioCfgScaleExtra, audio->cfg_scale); + audio->stg_scale = ExtraDouble(extras, kLtx2AudioStgScaleExtra, audio->stg_scale); + audio->rescale_scale = ExtraDouble(extras, kLtx2AudioRescaleScaleExtra, audio->rescale_scale); + audio->modality_scale = ExtraDouble(extras, kLtx2V2aGuidanceScaleExtra, audio->modality_scale); + audio->skip_step = ExtraInt(extras, kLtx2AudioSkipStepExtra, audio->skip_step); + ApplyStgBlocksExtra(extras, kLtx2AudioStgBlocksExtra, &audio->stg_blocks); + + const auto check_skip = [](const char* key, int64_t value) { + if (value >= 0) return; + Fail("'" + std::string(key) + "' is " + std::to_string(value) + + "; `should_skip_step` is `step % (skip_step + 1)` (guiders.py:287-291) and a negative " + "value would take the modulus of a non-positive divisor"); + }; + check_skip(kLtx2VideoSkipStepExtra, video->skip_step); + check_skip(kLtx2AudioSkipStepExtra, audio->skip_step); + // AN EMPTY LIST IS NOT REFUSED, and this function refused it until 2026-08-17. + // + // The refusal read: an empty `stg_blocks` beside a non-zero STG scale is a + // perturbed pass identical to the conditional one, so it is a wasted forward + // and a guidance term of exactly zero. Every clause of that is true and none + // of it makes the configuration illegal upstream, which is the only question + // a mirror gets to ask. Measured at Lightricks/LTX-2 `fd4ded7f`: + // + // - `packages/ltx-pipelines/docs/multimodal-guidance.md:13` documents it as + // THE way to turn STG off: "Set to `[]` to disable STG", in the same table + // and the same idiom as `stg_scale` -> 0.0 and `cfg_scale` -> 1.0. + // - `MultiModalGuiderParams.stg_blocks` DEFAULTS to `[]` + // (guiders.py:204, `field(default_factory=list)`). + // - `--video-stg-blocks` / `--audio-stg-blocks` are `nargs="*"` + // (args.py:979-985, :1039-1045, :1107-1113), so the flag with zero values + // parses to `[]`. `nargs="+"` was the one-character way to forbid it. + // - `LTX_2_3_HQ_PARAMS` SHIPS `stg_blocks=[]` on both modalities + // (constants.py:105, :113). + // - There is no validation of `stg_blocks` anywhere in that tree: no + // emptiness check, no length check, no range check against the block + // count. + // + // Upstream's semantics are unambiguous and are the reason `[]` is meaningful: + // `blocks=None` means EVERY block and `blocks=[]` means NO block + // (perturbations.py:26-33). The empty list is how a caller says the second + // thing, and `ApplyStgBlocksExtra` above exists to keep PRESENT-and-empty + // distinct from ABSENT for exactly that reason. Refusing it here made that + // distinction unreachable. + // + // WHAT IS STILL REFUSED, one layer down in `Ltx2GuidedDenoise`: a list that + // NAMES blocks and reaches none of them, e.g. `[28]` on a two-block DiT. That + // is a local condition rather than an upstream one — upstream only ever runs + // 48-block checkpoints and this port runs reduced ones — and it is a mismatch + // between a request and a checkpoint rather than an expressed intent. +} + +// Everything step 0 of phase 0 produced, for the gate that decides WHICH SPACE +// each arm was combined in. Derived at the call from what the seam returned, so +// a mutation to any arm moves a recorded field rather than leaving a comment +// that compiles. +void RecordFirstGuidedStep(Ltx2ConditioningTrace* trace, const Ltx2GuidedDenoiseResult& guided, + const std::vector& latent, + const std::vector& timesteps, double sigma, + const std::vector& stepper_input) { + const auto slot = [](Ltx2DenoisePass pass) { return static_cast(pass); }; + trace->video_guided = true; + trace->video_cond_forwards = guided.pass_ran[slot(Ltx2DenoisePass::kCond)] ? 1 : 0; + trace->video_uncond_forwards = guided.pass_ran[slot(Ltx2DenoisePass::kUncond)] ? 1 : 0; + trace->video_perturbed_forwards = guided.pass_ran[slot(Ltx2DenoisePass::kPerturbed)] ? 1 : 0; + trace->video_modality_forwards = guided.pass_ran[slot(Ltx2DenoisePass::kModality)] ? 1 : 0; + trace->video_perturbed_blocks = guided.perturbed_video_blocks; + trace->video_audio_perturbed_blocks = guided.perturbed_audio_blocks; + trace->video_modality_skipped_a2v = guided.modality_pass_skipped_a2v; + trace->video_modality_skipped_v2a = guided.modality_pass_skipped_v2a; + trace->video_first_latent = latent; + trace->video_first_timesteps = timesteps; + trace->video_first_cond = guided.video_pass[slot(Ltx2DenoisePass::kCond)]; + trace->video_first_cond_velocity = guided.video_pass_velocity[slot(Ltx2DenoisePass::kCond)]; + trace->video_first_uncond = guided.video_pass[slot(Ltx2DenoisePass::kUncond)]; + trace->video_first_uncond_velocity = guided.video_pass_velocity[slot(Ltx2DenoisePass::kUncond)]; + trace->video_first_perturbed = guided.video_pass[slot(Ltx2DenoisePass::kPerturbed)]; + trace->video_first_perturbed_velocity = + guided.video_pass_velocity[slot(Ltx2DenoisePass::kPerturbed)]; + trace->video_first_modality = guided.video_pass[slot(Ltx2DenoisePass::kModality)]; + trace->video_first_modality_velocity = + guided.video_pass_velocity[slot(Ltx2DenoisePass::kModality)]; + trace->video_first_denoised = guided.video_denoised; + trace->video_first_stepper_input = stepper_input; + trace->video_first_sigma = sigma; +} + } // namespace VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { @@ -1467,7 +1679,17 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { kv.first == kLtx2AudioStgScaleExtra || kv.first == kLtx2AudioRescaleScaleExtra || kv.first == kLtx2AudioSkipStepExtra || - kv.first == kLtx2AudioStgBlocksExtra; + kv.first == kLtx2AudioStgBlocksExtra || + // The VIDEO guider's row (row LTX25-GUIDED-VIDEO, #1092), + // from the same parser as the audio row above + // (utils/args.py:947-1066). + kv.first == kLtx2VideoCfgScaleExtra || + kv.first == kLtx2VideoStgScaleExtra || + kv.first == kLtx2VideoRescaleScaleExtra || + kv.first == kLtx2VideoSkipStepExtra || + kv.first == kLtx2VideoStgBlocksExtra || + kv.first == kLtx2A2vGuidanceScaleExtra || + kv.first == kLtx2V2aGuidanceScaleExtra; if (!known) { Fail("unknown per-generation extra '" + kv.first + "'. This family defines: " + std::string(kLtx2ImageCrfExtra) + ", " + kLtx2AudioPathExtra + ", " + @@ -1478,41 +1700,50 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { kLtx2RegenerateAudioExtra + ", " + kLtx2NegativePromptExtra + ", " + kLtx2AudioCfgScaleExtra + ", " + kLtx2AudioStgScaleExtra + ", " + kLtx2AudioRescaleScaleExtra + ", " + kLtx2AudioSkipStepExtra + ", " + - kLtx2AudioStgBlocksExtra); + kLtx2AudioStgBlocksExtra + ", " + kLtx2VideoCfgScaleExtra + ", " + + kLtx2VideoStgScaleExtra + ", " + kLtx2VideoRescaleScaleExtra + ", " + + kLtx2VideoSkipStepExtra + ", " + kLtx2VideoStgBlocksExtra + ", " + + kLtx2A2vGuidanceScaleExtra + ", " + kLtx2V2aGuidanceScaleExtra); } } - // ── the TEXT-TO-AUDIO knobs belong to ONE pipeline (#1005) ──────────────── + // ── the knobs that belong to ONE pipeline (#1005, corrected by #1092) ───── // // `pipeline_kind` is a LOAD extra, so which pipeline runs is settled before a - // request arrives and this is a decidable question rather than a guess. The - // guard runs in BOTH directions: the six T2A knobs are refused off a - // `t2a_one_stage` engine, and every other per-generation knob is refused ON - // one. Neither is padding. Upstream's `T2AOneStagePipeline.__call__` takes no - // image, no reference, no keyframe and no window (t2a_one_stage.py:109-122), - // and its guider arguments have no counterpart in any other `__call__`, so a - // knob crossing either way would silently do nothing to a render that still - // finishes. + // request arrives and this is a decidable question rather than a guess. + // + // WHAT #1092 CORRECTED, and why the old list was defensible until it was not. + // Row LTX25-T2A-ONE-STAGE refused `negative_prompt` and the five `audio_*` + // guider knobs on ANY non-t2a engine, reasoning that "no other pipeline + // `__call__` upstream takes a guider argument at all". That sentence was + // FALSE about upstream and TRUE about this port. Upstream's + // `default_1_stage_arg_parser` carries `--negative-prompt` + // (utils/args.py:937-946) and the whole audio guider row + // (`:1011-1075`) alongside the video one, and `TI2VidOneStagePipeline` + // consumes both through `audio_guider_params` (ti2vid_one_stage.py:215-218). + // What made the refusal harmless was that NOTHING HERE READ THEM on a joint + // render — the video denoise loop was unguided. Row LTX25-GUIDED-VIDEO makes + // them live, so the refusal would now reject a flag upstream serves. + // + // The guard therefore keeps one direction and drops the other: the knobs that + // describe a PICTURE are refused on a text-to-audio engine, which produces + // none. `T2AOneStagePipeline.__call__` takes a prompt, a negative prompt, a + // seed, a frame rate, a step count, the audio guider and a frame count + // (t2a_one_stage.py:109-122) and nothing else. { - const char* const kT2aOnly[] = {kLtx2NegativePromptExtra, kLtx2AudioCfgScaleExtra, - kLtx2AudioStgScaleExtra, kLtx2AudioRescaleScaleExtra, - kLtx2AudioSkipStepExtra, kLtx2AudioStgBlocksExtra}; const char* const kNotOnT2a[] = {kLtx2ImageCrfExtra, kLtx2AudioPathExtra, kLtx2AudioStartTimeExtra, kLtx2AudioMaxDurationExtra, kLtx2GeneratedKeyframesExtra, kLtx2TemporalRoundsExtra, kLtx2RetakeStartTimeExtra, kLtx2RetakeEndTimeExtra, kLtx2RetakeFrameRateExtra, kLtx2RegenerateVideoExtra, - kLtx2RegenerateAudioExtra}; - for (const char* key : kT2aOnly) { - if (!im.recipe.audio_only && !VideoExtra(gen.extras, key).empty()) { - Fail("the '" + std::string(key) + - "' extra is text-to-audio's alone (ltx-pipelines utils/args.py:1083-1119, the " - "`default_1_stage_t2a_arg_parser`), and this engine was loaded with pipeline_kind '" + - im.pipeline_kind + - "'. Refused rather than ignored: no other pipeline `__call__` upstream takes a " - "guider argument at all, so accepting it here would report a configured render " - "that ran the recipe's own values"); - } - } + kLtx2RegenerateAudioExtra, + // The VIDEO guider's own row: there is no + // video stream to guide, and upstream's t2a + // parser exposes none of them + // (utils/args.py:1083-1119). + kLtx2VideoCfgScaleExtra, kLtx2VideoStgScaleExtra, + kLtx2VideoRescaleScaleExtra, kLtx2VideoSkipStepExtra, + kLtx2VideoStgBlocksExtra, kLtx2A2vGuidanceScaleExtra, + kLtx2V2aGuidanceScaleExtra}; for (const char* key : kNotOnT2a) { if (im.recipe.audio_only && !VideoExtra(gen.extras, key).empty()) { Fail("the '" + std::string(key) + @@ -2323,6 +2554,145 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { im.trace.retake_latent_absmax = AbsMax(retake_video_volume); } + // ── THE GUIDERS, and the negative conditioning they ask for (#1092) ─────── + // + // `create_multimodal_guider_factory(params=..., negative_context=...)` once per + // stream, before the stage runs (ti2vid_one_stage.py:210-218). Resolved for + // EVERY phase up front rather than inside the loop, because the negative + // encode below is a host-side pass over the text tower and must happen once + // for the whole render if ANY phase asks for it. + // + // A phase whose recipe sets no guidance keeps `Ltx2MultiModalGuiderParams`'s + // own defaults — `cfg 1.0 / stg 0.0 / modality 1.0 / rescale 0.0` — which is + // exactly `_POSITIVE_ONLY_GUIDER` (denoisers.py:25-28). Only `OneStagePhase` + // sets real scales, so `distilled_two_stage`, `dfr`, `retake` and `dmd2` run + // ONE forward per step through the guided seam and combine it with a guider + // whose every term is zero, which is `SimpleDenoiser`'s output. Upstream + // selects `SimpleDenoiser` by PIPELINE (distilled.py:266,295) rather than by + // params; the two agree here because the recipes that select it are exactly + // the recipes whose guidance is the no-op one. + struct PhaseGuidance { + Ltx2MultiModalGuiderParams video; + Ltx2MultiModalGuiderParams audio; + }; + std::vector phase_guidance(recipe.phases.size()); + bool wants_negative = false; + bool wants_perturbation = false; + for (size_t p = 0; p < recipe.phases.size(); ++p) { + phase_guidance[p].video = recipe.phases[p].video_guidance; + phase_guidance[p].audio = recipe.phases[p].audio_guidance; + ApplyGuidanceOverrides(gen.extras, recipe.phases[p], &phase_guidance[p].video, + &phase_guidance[p].audio); + if (phase_guidance[p].video.DoUnconditionalGeneration() || + phase_guidance[p].audio.DoUnconditionalGeneration()) { + wants_negative = true; + } + if (phase_guidance[p].video.DoPerturbedGeneration() || + phase_guidance[p].audio.DoPerturbedGeneration() || + phase_guidance[p].video.DoIsolatedModalityGeneration() || + phase_guidance[p].audio.DoIsolatedModalityGeneration()) { + wants_perturbation = true; + } + } + + // REFUSED BY NAME, not degraded. `Ltx2DitForwardDevice` (ltx2_device.h:136) + // takes no `perturbations` argument, so the perturbed and isolated-modality + // passes on the device arm would run an UNPERTURBED forward — a finite clip + // whose `stg_scale * (cond - perturbed)` and `(modality_scale - 1) * (cond - + // mod)` terms are identically zero, and which is indistinguishable from a + // working render at every output this engine has. Classifier-free guidance + // alone is a different CONTEXT and no perturbation, so it is served on both + // arms. + if (im.on_device && wants_perturbation) { + Fail("this render's guidance needs a PERTURBED forward (STG, or the isolated-modality pass " + "that `modality_scale != 1.0` selects) and `Ltx2DitForwardDevice` takes no " + "`perturbations` argument, so the device-resident arm cannot run one. Refusing rather " + "than running an unperturbed forward, which would leave the STG and modality terms " + "exactly zero and render. Set '" + + std::string(kLtx2VideoStgScaleExtra) + "' and '" + + std::string(kLtx2AudioStgScaleExtra) + "' to 0.0 and '" + + std::string(kLtx2A2vGuidanceScaleExtra) + "' and '" + + std::string(kLtx2V2aGuidanceScaleExtra) + + "' to 1.0 to run classifier-free guidance alone on this arm, or load with device 0. " + "Owed by row LTX25-GUIDED-VIDEO (#1092)."); + } + + // The second half of upstream's ONE `PromptEncoder` call over + // `[prompt, negative_prompt]` (ti2vid_one_stage.py:166-174). Encoded ONLY when + // a guider asks: `do_unconditional_generation` is `not isclose(cfg_scale, 1.0)` + // (guiders.py:275-277), and at 1.0 there is no unconditional forward, so + // encoding it would be a wasted host-side 12B pass per request. + std::vector negative_video, negative_audio; + const float* negative_video_context = nullptr; + const float* negative_audio_context = nullptr; + if (wants_negative) { + if (!im.negative_video_prompt_embeds.empty() && gen.prompt.empty()) { + // The embeds fallback's own second half. Taken only when the request + // carries no prompt, which is the same polarity the POSITIVE fallback has + // above: a typed prompt encodes both halves through the tower. + if (im.prompt_tokens != context_tokens) { + Fail("the negative prompt embeds hold " + std::to_string(im.prompt_tokens) + + " rows and this request's conditioning holds " + std::to_string(context_tokens) + + "; the guidance delta would subtract tensors that do not correspond"); + } + negative_video_context = im.negative_video_prompt_embeds.data(); + negative_audio_context = im.negative_audio_prompt_embeds.data(); + } else if (!im.has_encoder) { + Fail("this render needs an unconditional forward (the video cfg scale is " + + std::to_string(phase_guidance[0].video.cfg_scale) + " and the audio one is " + + std::to_string(phase_guidance[0].audio.cfg_scale) + + "), which needs the NEGATIVE prompt encoded — and no text tower is loaded. The " + "positive `prompt_embeds_path` fallback carries ONE conditioning pair; supply the " + "second through '" + + std::string(kLtx2NegativePromptEmbedsExtra) + "' and '" + + std::string(kLtx2NegativeAudioPromptEmbedsExtra) + + "', load with encoder_path, or set '" + std::string(kLtx2VideoCfgScaleExtra) + + "' and '" + std::string(kLtx2AudioCfgScaleExtra) + + "' to 1.0, which turns the unconditional pass off (guiders.py:275-277)"); + } else { + const std::string negative = + VideoExtra(gen.extras, kLtx2NegativePromptExtra, recipe.negative_prompt); + if (negative.empty()) { + Fail("this render needs a negative prompt and neither the '" + + std::string(kLtx2NegativePromptExtra) + + "' extra nor the recipe carries one. An EMPTY negative prompt is not the same as no " + "CFG: it still encodes and still steers, and upstream's CLI always supplies " + "`DEFAULT_NEGATIVE_PROMPT` (utils/args.py:937-946)"); + } + if (!recipe.allow_negative_prompt) { + Fail("this recipe takes no negative prompt (`prompts_to_encode` is `[prompt]` alone), so " + "a guider asking for the unconditional forward is a contradiction rather than a " + "request this engine can serve"); + } + vt::Queue text_queue{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + const Ltx2PromptConditioning encoded = Ltx2EncodePromptToConditioning( + *im.tower, *im.tokenizer, im.gemma_ids, im.caption_projections, im.feature_cfg, + negative, text_queue); + negative_video = encoded.conditioning.video; + negative_audio = encoded.conditioning.audio; + if (encoded.seq != context_tokens) { + // Upstream's two encodings come from ONE tokenization of a two-element + // list, so they share a padded width by construction. A mismatch means + // the two ran different geometries and the guidance delta would subtract + // tensors that do not correspond. + Fail("the negative prompt encoded to " + std::to_string(encoded.seq) + + " context rows and the prompt to " + std::to_string(context_tokens) + + "; upstream encodes both in one call and they cannot differ"); + } + if (im.has_connector) { + const Ltx2ConnectorEmbeddings through = + RunConnector(SafetensorsFile::Open(im.params.dit_path), im.video_connector_cfg, + im.audio_connector_cfg, encoded.conditioning.video, + encoded.conditioning.audio, encoded.conditioning.additive_mask, + context_tokens); + negative_video = through.video; + negative_audio = through.audio; + } + negative_video_context = negative_video.data(); + negative_audio_context = negative_audio.data(); + } + } + for (int64_t phase_index = 0; phase_index <= last_phase; ++phase_index) { const Ltx2PhaseRecipe& phase = recipe.phases[static_cast(phase_index)]; const int64_t phase_h = height / phase.spatial_downscale; @@ -2933,6 +3303,27 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { // from the state noise, so its first draw is not the initial latent's. SplitMixGaussian loop_noise(seed + static_cast(phase.noise_seed_offset)); const int64_t sigma_count = static_cast(sigmas.size()); + + // This phase's two guiders, resolved once. `GuidedDenoiser` is constructed + // per stage upstream and holds its guiders for the whole loop + // (ti2vid_one_stage.py:221-226), so resolving them per step would let a + // request override change meaning halfway down a schedule. + const Ltx2MultiModalGuiderParams& video_guidance = + phase_guidance[static_cast(phase_index)].video; + const Ltx2MultiModalGuiderParams& audio_guidance = + phase_guidance[static_cast(phase_index)].audio; + if (phase_index == 0) { + im.trace.video_guidance_cfg_scale = video_guidance.cfg_scale; + im.trace.video_guidance_stg_scale = video_guidance.stg_scale; + im.trace.video_guidance_rescale_scale = video_guidance.rescale_scale; + im.trace.video_guidance_modality_scale = video_guidance.modality_scale; + } + + // `_last_denoised_video` / `_last_denoised_audio` (denoisers.py:274-275): + // per DENOISER, so per phase, and empty until the first step fills them. A + // skipped step reuses them instead of running a forward. + std::vector last_denoised_video; + std::vector last_denoised_audio; for (int64_t step = 0; step + 1 < sigma_count; ++step) { const float sigma = sigmas[static_cast(step)]; const std::vector v_timesteps = TimestepsFromMask(video, sigma); @@ -3029,20 +3420,87 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { ain.positions = audio.positions.data(); ain.context = audio_context; + // ── the X0 MODEL (model.py:590-604), and the guided denoiser ────────── + // + // `DiffusionStage` never hands the loop the raw velocity model: it hands + // `X0Model(builder.build(...))` (utils/blocks.py:480-482). So `to_denoised` + // belongs HERE, inside the wrapper, applied to EVERY pass on its way out of + // the forward — and the guider downstream combines already-denoised + // tensors. Converting once after the guider instead is a DIFFERENT function + // wherever `rescale_scale != 0` (guiders.py:268-271), which is 0.7 on every + // video row of the params table. That defect shipped on the audio arm of + // this tree and is #1039. + // // One graph, two residencies. On the CPU this is the L2 parity forward in // its declared f32; on an accelerator it is the phase-L8 device-resident // forward over the bf16 the DiT was STAGED at, and the two agree on - // everything but where the bytes live and how wide they are. - const Ltx2DitOutputs velocity = - im.on_device ? Ltx2DitForwardDevice(*im.queue, im.dit.params, im.dit.weights, &vin, - &ain, im.compute_dtype) - : Ltx2DitForward(im.device, im.dit.params, im.dit.weights, &vin, &ain, - im.compute_dtype); - - const std::vector v_denoised = PostProcessLatent( - ToDenoised(video.latent, velocity.video, v_timesteps, video.tokens, video.width), video); - const std::vector a_denoised = PostProcessLatent( - ToDenoised(audio.latent, velocity.audio, a_timesteps, audio.tokens, audio.width), audio); + // everything but where the bytes live and how wide they are. The device + // forward takes no `perturbations`, which is why a guider that asks for the + // perturbed or isolated-modality pass on that arm is refused before the + // loop rather than served an unperturbed forward. + const Ltx2X0Model x0_model = [&](const Ltx2ModalityInput* v, const Ltx2ModalityInput* a, + const Ltx2DitPerturbation* p) { + // The refusal above is a statement about the RECIPE; this is a statement + // about the CALL, and the two are not the same check. A pass that reached + // here with a perturbation on the device arm would have it silently + // dropped by the argument list below, which is the shape of defect this + // file keeps finding: correct output for the wrong reason, with the STG + // and modality terms at exactly zero and nothing in the frames, the + // shapes or the counts to show for it. + VT_CHECK(!im.on_device || p == nullptr, + "ltx2 video: a perturbed forward reached the device-resident arm, where " + "`Ltx2DitForwardDevice` has no `perturbations` argument to take it. The guidance " + "resolution refuses this before the loop, so reaching it is a defect rather than " + "a bad request. Owed by row LTX25-GUIDED-VIDEO (#1092)."); + const Ltx2DitOutputs velocity = + im.on_device ? Ltx2DitForwardDevice(*im.queue, im.dit.params, im.dit.weights, v, a, + im.compute_dtype) + : Ltx2DitForward(im.device, im.dit.params, im.dit.weights, v, a, + im.compute_dtype, /*cache=*/nullptr, p); + Ltx2X0Outputs out; + out.video_velocity = velocity.video; + out.audio_velocity = velocity.audio; + // The PER-TOKEN timesteps, not the schedule scalar: a conditioned token + // sits at timestep 0 and using the scalar there re-noises it. + out.video = + ToDenoised(video.latent, velocity.video, v_timesteps, video.tokens, video.width); + out.audio = + ToDenoised(audio.latent, velocity.audio, a_timesteps, audio.tokens, audio.width); + return out; + }; + + Ltx2GuidedDenoiseInputs denoise_in; + denoise_in.video = &vin; + denoise_in.audio = &ain; + denoise_in.video_negative_context = negative_video_context; + denoise_in.audio_negative_context = negative_audio_context; + denoise_in.video_guider = video_guidance; + denoise_in.audio_guider = audio_guidance; + denoise_in.num_blocks = im.dit.params.num_layers; + denoise_in.step_index = step; + denoise_in.last_denoised_video = &last_denoised_video; + denoise_in.last_denoised_audio = &last_denoised_audio; + const Ltx2GuidedDenoiseResult guided = Ltx2GuidedDenoise(x0_model, denoise_in); + + // `post_process_latent` is applied by the LOOP to the guider's OUTPUT + // (samplers.py:35, :484), never per arm inside the denoiser. Pinning the + // conditioned tokens per arm would make every arm agree on exactly those + // tokens, which zeroes the guidance delta precisely where a keyframe or a + // reference clip is conditioning — a render that is correct everywhere the + // conditioning is absent. + // + // `last_denoised_*` keeps what the GUIDER returned, before the + // post-process, because that is what `_last_denoised_video` holds + // (denoisers.py:299-300) and what a skipped step reuses. + last_denoised_video = guided.video_denoised; + last_denoised_audio = guided.audio_denoised; + const std::vector v_denoised = PostProcessLatent(guided.video_denoised, video); + const std::vector a_denoised = PostProcessLatent(guided.audio_denoised, audio); + + if (phase_index == 0 && step == 0) { + RecordFirstGuidedStep(&im.trace, guided, video.latent, v_timesteps, + static_cast(sigma), v_denoised); + } const bool terminal = sigmas[static_cast(step + 1)] == 0.0F; if (phase.stepper == Ltx2StepperKind::kEulerAncestral) { @@ -3051,6 +3509,7 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { // taking an ancestral step there would re-noise the finished latent. video.latent = v_denoised; audio.latent = a_denoised; + if (phase_index == 0 && step == 0) im.trace.video_first_next_latent = video.latent; continue; } const std::vector v_noise = @@ -3075,6 +3534,11 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { sigma_count, step, static_cast(audio.latent.size())); } + // What the sampler WROTE, recorded after the step rather than derived from + // what was recorded before it. It is the only observable that says which + // tensor the stepper was actually handed: a second `ToDenoised` on the way + // in leaves every other recorded field untouched. + if (phase_index == 0 && step == 0) im.trace.video_first_next_latent = video.latent; } // `clear_conditioning` + `unpatchify` (blocks.py:575-580, in that order). diff --git a/tests/vllm/multimodal/ltx2_video_fixture.h b/tests/vllm/multimodal/ltx2_video_fixture.h index 37fc7e93f..7306dc501 100644 --- a/tests/vllm/multimodal/ltx2_video_fixture.h +++ b/tests/vllm/multimodal/ltx2_video_fixture.h @@ -1425,6 +1425,12 @@ inline void WritePromptEmbeds(const std::string& path, const std::string& tag, i // The whole set, as an engine would be pointed at it. struct Paths { std::string dit, video_vae, audio_vae, upsampler, video_embeds, audio_embeds; + // The NEGATIVE half of the embeds fallback (row LTX25-GUIDED-VIDEO, #1092). + // Written from DIFFERENT tags than the positive pair, deliberately: a negative + // conditioning equal to the positive one makes `cond - uncond` identically + // zero, so the classifier-free term would vanish and every assertion about it + // would pass for the wrong reason. + std::string negative_video_embeds, negative_audio_embeds; // Phase L13: the text tower, and the Gemma config the shipped encoder does not // carry. Written by every fixture; POINTING the engine at them is opt-in, // because a load that materializes a tower is not what most cases here gate. @@ -1446,6 +1452,8 @@ inline Paths WriteFixture(const std::string& dir, int64_t prompt_tokens = 4) { p.upsampler = dir + "/upsampler.safetensors"; p.video_embeds = dir + "/video_prompt_embeds.f32"; p.audio_embeds = dir + "/audio_prompt_embeds.f32"; + p.negative_video_embeds = dir + "/negative_video_prompt_embeds.f32"; + p.negative_audio_embeds = dir + "/negative_audio_prompt_embeds.f32"; p.encoder = dir + "/text_encoder.safetensors"; p.encoder_config = dir + "/gemma_config.json"; WriteReducedTextEncoder(dit, p.encoder); @@ -1457,6 +1465,12 @@ inline Paths WriteFixture(const std::string& dir, int64_t prompt_tokens = 4) { WritePromptEmbeds(p.video_embeds, "ltx2.embeds.video", prompt_tokens, dit.cross_attention_dim); WritePromptEmbeds(p.audio_embeds, "ltx2.embeds.audio", prompt_tokens, dit.audio_cross_attention_dim); + // `.negative` tags, so the two halves differ. `Param` seeds from the NAME, so + // these are as deterministic as the positive pair and independent of it. + WritePromptEmbeds(p.negative_video_embeds, "ltx2.embeds.video.negative", prompt_tokens, + dit.cross_attention_dim); + WritePromptEmbeds(p.negative_audio_embeds, "ltx2.embeds.audio.negative", prompt_tokens, + dit.audio_cross_attention_dim); return p; } diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index 9c11de380..09fb59f9c 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -84,10 +84,38 @@ vllm::multimodal::VideoModelParams FixtureParams(const ltx2_fixture::Paths& path mp.audio_vae_path = paths.audio_vae; mp.prompt_embeds_path = paths.video_embeds; mp.extras[vllm::multimodal::kLtx2AudioPromptEmbedsExtra] = paths.audio_embeds; + // The NEGATIVE half (row LTX25-GUIDED-VIDEO, #1092), on the SHARED helper + // rather than only on the cases that gate guidance. A `one_stage` engine's own + // recipe resolves `cfg_scale = 3.0`, so upstream's unconditional forward is + // this fixture's DEFAULT configuration, and an engine that could not run it + // would make every one_stage case here a refusal. `distilled_two_stage` — the + // default kind, and what most cases below load — resolves `cfg_scale = 1.0` + // and never reads these. + mp.extras[vllm::multimodal::kLtx2NegativePromptEmbedsExtra] = paths.negative_video_embeds; + mp.extras[vllm::multimodal::kLtx2NegativeAudioPromptEmbedsExtra] = paths.negative_audio_embeds; mp.device = 0; return mp; } +// The ONE guider field a `one_stage` render on this fixture has to override, and +// the reason is the fixture rather than the row: the reduced DiT has TWO blocks +// (ltx2_video_fixture.h `ReducedDitParams`), so the params table's own +// `stg_blocks = [28]` (utils/constants.py:83-88) names a block this checkpoint +// does not have. Left alone, the perturbed forward would perturb nothing and +// `stg_scale * (cond - perturbed)` would be exactly zero — which the engine now +// refuses by name rather than rendering, so this is what turns that refusal into +// a render. Named explicitly rather than by setting the STG scale to 0, because +// turning the perturbed pass OFF is a different configuration and would vacate +// every assertion about it. +// +// Nothing else is overridden: `cfg_scale`, `rescale_scale` and `modality_scale` +// stay at the recipe's own 3.0 / 0.7 / 3.0, which is what makes the guided cases +// below sit on the DEFAULT arm. +void OneStageFixtureGuidance(vllm::multimodal::VideoGenParams* gen) { + gen->extras[vllm::multimodal::kLtx2VideoStgBlocksExtra] = "1"; + gen->extras[vllm::multimodal::kLtx2AudioStgBlocksExtra] = "1"; +} + // The smallest request the fixture's own scale factors admit: (8, 32, 32) means // 64x64 pixels is a 2x2 latent and 9 frames is 2 latent frames. vllm::multimodal::VideoGenParams FixtureGen(const std::string& out_dir) { @@ -755,6 +783,7 @@ TEST_CASE("ltx2 video: a size that does not divide the latent grid is REFUSED, p vllm::multimodal::LoadVideoEngine(mp); vllm::multimodal::VideoGenParams gen = FixtureGen(ws.root + "/one_stage_96"); gen.width = 96; + OneStageFixtureGuidance(&gen); const vllm::multimodal::VideoResult result = engine->Generate(gen); CHECK(result.width == 96); CHECK(result.height == 64); @@ -954,6 +983,12 @@ TEST_CASE("ltx2 video: every accepted load extra is READ by something") { // have readers -- `lora_path` builds an `Ltx2LoraSpec` and `lora_strength` // is parsed into it -- so they belong here and not in `refused`. vllm::multimodal::kLtx2LoraPathExtra, vllm::multimodal::kLtx2LoraStrengthExtra, + // Row LTX25-GUIDED-VIDEO (#1092): the NEGATIVE half of the embeds + // fallback. Both are read where the positive pair is, and both are read + // again by the guided denoise loop when a guider asks for the + // unconditional forward. + vllm::multimodal::kLtx2NegativePromptEmbedsExtra, + vllm::multimodal::kLtx2NegativeAudioPromptEmbedsExtra, }; // The keys the family defines and does NOT serve. Growing this list is a // deliberate act; growing it silently is the defect #611 records. @@ -981,8 +1016,8 @@ TEST_CASE("ltx2 video: every accepted load extra is READ by something") { // Every name this row inventoried is still accepted... for (const std::string& key : served) CHECK(listing.find(key) != std::string::npos); for (const std::string& key : refused) CHECK(listing.find(key) != std::string::npos); - // ...and there is no THIRTEENTH name that this inventory has never seen. The - // separator is ", ", so the count is one more than the separators. + // ...and there is no name past the end of this inventory that it has never + // seen. The separator is ", ", so the count is one more than the separators. size_t names = 1; for (size_t at = listing.find(", "); at != std::string::npos; at = listing.find(", ", at + 2)) { ++names; @@ -1084,14 +1119,16 @@ TEST_CASE("ltx2 video: the recorded reader anchors are the ones in the source") } REQUIRE(array_end > array_line); - // The eleven SERVED keys, by the token each is spelled with in the source. Order - // is irrelevant — the comparison is on the sorted set — so this list is not a - // second place the anchors live. + // The thirteen SERVED keys, by the token each is spelled with in the source. + // Order is irrelevant — the comparison is on the sorted set — so this list is + // not a second place the anchors live. The last two arrived with row + // LTX25-GUIDED-VIDEO (#1092). const std::vector served_tokens = { "kLtx2AudioPromptEmbedsExtra", "kLtx2PipelineKindExtra", "kLtx2ModelVersionExtra", "kLtx2AllowUnportedExtra", "kLtx2MaxPhaseExtra", "kLtx2DitConfigPathExtra", "kLtx2PromptValidRowsExtra", "kLtx2EncoderConfigPathExtra", "\"upsampler_path\"", "kLtx2LoraPathExtra", "kLtx2LoraStrengthExtra", + "kLtx2NegativePromptEmbedsExtra", "kLtx2NegativeAudioPromptEmbedsExtra", }; std::vector derived; for (const std::string& token : served_tokens) { @@ -3180,6 +3217,7 @@ TEST_CASE("ltx2 video: a LAST-frame keyframe is APPENDED, and the sequence is tr vllm::multimodal::VideoGenParams gen = request("one_stage_kf", kf_a_path); gen.steps = 2; // one_stage admits a step override; 50 would gate nothing extra + OneStageFixtureGuidance(&gen); (void)engine->Generate(gen); const vllm::multimodal::Ltx2ConditioningTrace trace = ltx2->last_conditioning(); @@ -5057,26 +5095,82 @@ TEST_CASE("ltx2 t2a: the refusals name what is missing, and each is checked HERE } } -TEST_CASE("ltx2 t2a: a t2a-only knob is refused on the video pipelines") { +TEST_CASE("ltx2: the AUDIO guider knobs are NOT t2a-only, and this case used to say they were") { + // WHAT THIS CASE ASSERTED UNTIL ROW LTX25-GUIDED-VIDEO (#1092): that + // `audio_cfg_guidance_scale` on a video pipeline is refused "text-to-audio's + // alone". The premise behind that refusal was that "no other pipeline + // `__call__` upstream takes a guider argument at all", and it is FALSE about + // upstream: `default_1_stage_arg_parser` carries the whole audio guider row + // beside the video one (ltx-pipelines utils/args.py:1011-1075 @ fd4ded7f) and + // `TI2VidOneStagePipeline` builds `audio_guider_params` from it + // (ti2vid_one_stage.py:215-218). It was harmless only because the joint + // render here was unguided, so nothing could have read the knob. + // + // The correction is kept as an executable statement rather than a deletion, + // because "this used to be refused" is exactly what a later reader needs. Workspace ws; - const vllm::multimodal::VideoModelParams mp = EncoderParams(ws.paths); + // A `one_stage` engine with the fixture's own text tower, which is the + // configuration these knobs describe: `distilled_two_stage` fixes its guidance + // and refuses every override, so asking it would test the other guard. + vllm::multimodal::VideoModelParams mp = EncoderParams(ws.paths); + mp.extras[vllm::multimodal::kLtx2PipelineKindExtra] = "one_stage"; const std::unique_ptr engine = vllm::multimodal::LoadVideoEngine(mp); REQUIRE(engine != nullptr); - vllm::multimodal::VideoGenParams gen = PromptedGen(ws.root + "/x", "a b c"); - gen.extras[vllm::multimodal::kLtx2AudioCfgScaleExtra] = "5.0"; - try { + const auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + + // FIRST, the negative prompt through the TOWER, which is the path the embeds + // fallback exists to stand in for. The recipe's own default negative prompt is + // upstream's `DEFAULT_NEGATIVE_PROMPT` -- an English sentence -- and this + // fixture's tokenizer carries a three-token vocabulary, so it is overridden + // here exactly as `--negative-prompt` is for (utils/args.py:937-946). + { + vllm::multimodal::VideoGenParams gen = PromptedGen(ws.root + "/tower_negative", "a b c"); + gen.steps = 2; + OneStageFixtureGuidance(&gen); + gen.extras[vllm::multimodal::kLtx2NegativePromptExtra] = "c b a"; (void)engine->Generate(gen); - FAIL_CHECK("a t2a guider knob must be refused on a video pipeline"); + const vllm::multimodal::Ltx2ConditioningTrace t = ltx->last_conditioning(); + REQUIRE(t.completed); + CHECK(t.video_uncond_forwards == 1); + // And it was a DIFFERENT forward, so the tower's negative encoding reached + // the DiT rather than the positive one being handed over twice. + REQUIRE(!t.video_first_uncond.empty()); + CHECK(t.video_first_uncond != t.video_first_cond); + } + + // SECOND, the knob itself. At 1.0 on both streams there is no unconditional + // pass at all (guiders.py:275-277) and nothing to encode. Without this the + // case would pass on a build that accepted the knob and ignored it, which is + // the defect the extras surface exists to refuse. + { + vllm::multimodal::VideoGenParams gen = PromptedGen(ws.root + "/audio_knob", "a b c"); + gen.steps = 2; + OneStageFixtureGuidance(&gen); + gen.extras[vllm::multimodal::kLtx2AudioCfgScaleExtra] = "1.0"; + gen.extras[vllm::multimodal::kLtx2VideoCfgScaleExtra] = "1.0"; + (void)engine->Generate(gen); + const vllm::multimodal::Ltx2ConditioningTrace t = ltx->last_conditioning(); + CHECK(t.completed); + CHECK(t.video_uncond_forwards == 0); + } + + // The direction that SURVIVES: a knob that describes a picture, on an engine + // that renders none. + const std::unique_ptr t2a = + vllm::multimodal::LoadVideoEngine(T2aParams(ws.paths)); + REQUIRE(t2a != nullptr); + vllm::multimodal::VideoGenParams t2a_gen = T2aGen(ws.root + "/video_knob", "a b c"); + t2a_gen.extras[vllm::multimodal::kLtx2VideoCfgScaleExtra] = "5.0"; + try { + (void)t2a->Generate(t2a_gen); + FAIL_CHECK("a VIDEO guider knob must be refused on a text-to-audio engine"); } catch (const std::exception& e) { const std::string msg = e.what(); INFO(msg); - CHECK(msg.find("text-to-audio's alone") != std::string::npos); - // It names the pipeline the ENGINE resolved, so a message that guessed - // would say something else. - const auto* ltx = dynamic_cast(engine.get()); - REQUIRE(ltx != nullptr); - CHECK(msg.find("'" + ltx->pipeline_kind() + "'") != std::string::npos); + CHECK(msg.find("no meaning on a text-to-audio render") != std::string::npos); + CHECK(msg.find(vllm::multimodal::kLtx2VideoCfgScaleExtra) != std::string::npos); } } @@ -5158,6 +5252,202 @@ TEST_CASE("ltx2 t2a: the DiT forward runs ONE stream, and the old guard's reason vt::DType::kF32, /*cache=*/nullptr, &bad)); } +TEST_CASE("ltx2 dit: each CROSS perturbation gates ITS OWN direction and no other (#1092)") { + // WHY THIS CASE EXISTS, and it is a review mutation result rather than a + // symmetry a reader would ask for. + // + // Three mutations were run against the case above's sibling — the end-to-end + // `one_stage` guidance case — each built clean and each with its exit status + // captured directly: + // + // M12 the DiT ignores `video_cross_attn_skip_all` GREEN, exit 0 + // M13 the DiT ignores `audio_cross_attn_skip_all` GREEN, exit 0 + // M14 the DiT ignores BOTH RED, exit 1 + // M15 the DiT SWAPS which flag gates which direction GREEN, exit 0 + // + // A build that plumbs the flags and applies NEITHER was caught. A build that + // applies exactly one, or applies both to the wrong directions, was not — and + // the half-wrong build renders, on the DEFAULT video arm, whose + // `modality_scale` is 3.0. The end-to-end assertions cannot separate them: + // `MaxAbsDiffOf(video_first_modality, video_first_cond)` still fires with one + // direction applied, because the modality pass still differs from `cond`; and + // `Ltx2ConditioningTrace::video_modality_skipped_{a2v,v2a}` is assigned from + // the perturbation struct THE SEAM BUILT (ltx2_denoisers.cpp:315-316), which + // says what was handed over and nothing about what the DiT did with it. + // + // HOW ONE DIRECTION IS ISOLATED AT ALL, on a DiT with more than one block. + // Within a block the two directions are independent: both read the pre-cross + // snapshots `vx_pre` / `ax_pre` (transformer.py:333). ACROSS blocks they are + // not — block 1's V2A reads the video state block 0's A2V wrote — so a + // both-streams-enabled forward cannot attribute a change to a direction, and + // this fixture's DiT has two blocks. The separation therefore comes from + // upstream's own predicates (transformer.py:265-269): + // + // run_a2v = run_vx and audio is present run_vx = video.ENABLED and ... + // run_v2a = run_ax and video is present run_ax = audio.ENABLED and ... + // + // so `audio->enabled = false` with the audio stream still PRESENT runs A2V and + // not V2A, and `video->enabled = false` runs V2A and not A2V. That is the + // configuration `ltx2.h` already documents as rendering rather than failing, + // and it makes each direction observable alone. + Workspace ws; + const vllm::SafetensorsFile dit_file = vllm::SafetensorsFile::Open(ws.paths.dit); + vllm::Ltx2DitLoadOptions options; + options.widen_to_f32 = true; // `Ltx2DitForward` is f32 by declaration + const vllm::Ltx2DitCheckpoint ckpt = vllm::Ltx2LoadDitFromSafetensors(dit_file, options); + const vllm::Ltx2DitParams& params = ckpt.params; + + const int64_t video_tokens = 2, audio_tokens = 4, context_tokens = 3; + // VARYING PER TOKEN. A latent whose rows are all equal makes attention return + // a weighted average of identical values, which is the value projection again, + // and a perturbation that removes the whole branch would still be measurable — + // but the SELF-attention case above measured a constant fill turning its own + // assertion into a false negative, so the same discipline is applied here. + const auto fill = [](std::vector* v, float base) { + for (size_t i = 0; i < v->size(); ++i) { + (*v)[i] = base + 0.01F * static_cast(i % 7) - 0.02F * static_cast(i % 3); + } + }; + std::vector video_latent(static_cast(video_tokens * params.in_channels)); + std::vector audio_latent(static_cast(audio_tokens * params.audio_in_channels)); + fill(&video_latent, 0.25F); + fill(&audio_latent, 0.30F); + std::vector video_timesteps(static_cast(video_tokens), 0.5F); + std::vector audio_timesteps(static_cast(audio_tokens), 0.5F); + const float sigma = 0.5F; + std::vector video_positions(static_cast(3 * video_tokens * 2)); + std::vector audio_positions(static_cast(audio_tokens * 2)); + for (int64_t d = 0; d < 3; ++d) { + for (int64_t t = 0; t < video_tokens; ++t) { + video_positions[static_cast((d * video_tokens + t) * 2)] = + static_cast(t) * 0.04; + video_positions[static_cast((d * video_tokens + t) * 2 + 1)] = + static_cast(t + 1) * 0.04; + } + } + for (int64_t t = 0; t < audio_tokens; ++t) { + audio_positions[static_cast(t * 2)] = static_cast(t) * 0.04; + audio_positions[static_cast(t * 2 + 1)] = static_cast(t + 1) * 0.04; + } + std::vector video_context(static_cast(context_tokens * params.cross_attention_dim), + 0.05F); + std::vector audio_context( + static_cast(context_tokens * params.audio_cross_attention_dim), 0.07F); + + struct Streams { + vllm::Ltx2ModalityInput video; + vllm::Ltx2ModalityInput audio; + }; + const auto make = [&](bool video_enabled, bool audio_enabled) { + Streams s; + s.video.tokens = video_tokens; + s.video.context_tokens = context_tokens; + s.video.enabled = video_enabled; + s.video.latent = video_latent.data(); + s.video.timesteps = video_timesteps.data(); + s.video.sigma = σ + s.video.positions = video_positions.data(); + s.video.context = video_context.data(); + s.audio.tokens = audio_tokens; + s.audio.context_tokens = context_tokens; + s.audio.enabled = audio_enabled; + s.audio.latent = audio_latent.data(); + s.audio.timesteps = audio_timesteps.data(); + s.audio.sigma = σ + s.audio.positions = audio_positions.data(); + s.audio.context = audio_context.data(); + return s; + }; + const auto run = [&](Streams& io, const vllm::Ltx2DitPerturbation* p) { + return vllm::Ltx2DitForward(vt::Device{}, params, ckpt.weights, &io.video, &io.audio, + vt::DType::kF32, /*cache=*/nullptr, p); + }; + // `MaxAbsOf` / `MaxAbsDiffOf` are defined further down this file, after this + // case, so the two measurements are local rather than moved — moving them + // would churn a block three other cases read. + const auto max_abs = [](const std::vector& v) { + double m = 0.0; + for (const float x : v) m = std::max(m, std::abs(static_cast(x))); + return m; + }; + const auto moved = [](const std::vector& a, const std::vector& b) { + REQUIRE(a.size() == b.size()); + double m = 0.0; + for (size_t i = 0; i < a.size(); ++i) { + m = std::max(m, std::abs(static_cast(a[i]) - static_cast(b[i]))); + } + return m > 0.0; + }; + + vllm::Ltx2DitPerturbation skip_a2v; + skip_a2v.video_cross_attn_skip_all = true; // SKIP_A2V_CROSS_ATTN + vllm::Ltx2DitPerturbation skip_v2a; + skip_v2a.audio_cross_attn_skip_all = true; // SKIP_V2A_CROSS_ATTN + + SUBCASE("SKIP_A2V_CROSS_ATTN moves the VIDEO stream and SKIP_V2A does not") { + // `audio->enabled = false`, audio still present: A2V runs, V2A does not. + auto io = make(/*video_enabled=*/true, /*audio_enabled=*/false); + const vllm::Ltx2DitOutputs base = run(io, nullptr); + REQUIRE(base.video.size() == static_cast(video_tokens * params.out_channels)); + // NON-VACUITY. A zero output would make both comparisons below trivially + // equal, so the "did not move" half would pass on a forward that computed + // nothing at all. + REQUIRE(max_abs(base.video) > 1e-6); + + const vllm::Ltx2DitOutputs a2v_off = run(io, &skip_a2v); + REQUIRE(a2v_off.video.size() == base.video.size()); + CHECK_MESSAGE(moved(a2v_off.video, base.video), + "`video_cross_attn_skip_all` changed nothing on a forward where A2V is the only " + "cross direction running, so the flag reaches no guard " + "(transformer.py:335). The isolated-modality pass is then the conditional pass " + "again in the audio->video direction, on a recipe whose modality_scale is 3.0"); + + const vllm::Ltx2DitOutputs v2a_off = run(io, &skip_v2a); + REQUIRE(v2a_off.video.size() == base.video.size()); + CHECK_MESSAGE(v2a_off.video == base.video, + "`audio_cross_attn_skip_all` moved the VIDEO stream on a forward that runs no " + "V2A at all, so the two flags are wired to each other's directions. The flag " + "rides on the stream being WRITTEN (transformer.py:335, :367)"); + } + + SUBCASE("SKIP_V2A_CROSS_ATTN moves the AUDIO stream and SKIP_A2V does not") { + // `video->enabled = false`, video still present: V2A runs, A2V does not. + auto io = make(/*video_enabled=*/false, /*audio_enabled=*/true); + const vllm::Ltx2DitOutputs base = run(io, nullptr); + REQUIRE(base.audio.size() == static_cast(audio_tokens * params.audio_out_channels)); + REQUIRE(max_abs(base.audio) > 1e-6); + + const vllm::Ltx2DitOutputs v2a_off = run(io, &skip_v2a); + REQUIRE(v2a_off.audio.size() == base.audio.size()); + CHECK_MESSAGE(moved(v2a_off.audio, base.audio), + "`audio_cross_attn_skip_all` changed nothing on a forward where V2A is the only " + "cross direction running, so the flag reaches no guard (transformer.py:367)"); + + const vllm::Ltx2DitOutputs a2v_off = run(io, &skip_a2v); + REQUIRE(a2v_off.audio.size() == base.audio.size()); + CHECK_MESSAGE(a2v_off.audio == base.audio, + "`video_cross_attn_skip_all` moved the AUDIO stream on a forward that runs no " + "A2V at all, so the two flags are wired to each other's directions"); + } + + SUBCASE("the isolated-modality pass's OWN configuration moves both streams") { + // Both directions off with both streams enabled, which is what + // `_guided_denoise` builds for the `mod` pass (denoisers.py:125-138, + // `blocks=None` on both types). This is the shipped combination; the two + // subcases above are what separates its halves. + auto io = make(/*video_enabled=*/true, /*audio_enabled=*/true); + const vllm::Ltx2DitOutputs base = run(io, nullptr); + REQUIRE(max_abs(base.video) > 1e-6); + REQUIRE(max_abs(base.audio) > 1e-6); + vllm::Ltx2DitPerturbation both; + both.video_cross_attn_skip_all = true; + both.audio_cross_attn_skip_all = true; + const vllm::Ltx2DitOutputs off = run(io, &both); + CHECK(moved(off.video, base.video)); + CHECK(moved(off.audio, base.audio)); + } +} + TEST_CASE("ltx2 t2a: a SKIPPED step runs no forward and reuses the last prediction") { // `should_skip_step` is `step % (skip_step + 1) != 0` (guiders.py:287-291), so // `skip_step = 1` skips every ODD step. Upstream then returns @@ -5398,7 +5688,7 @@ TEST_CASE("ltx2 t2a: the guider is handed x0 predictions and not raw velocities" {"perturbed", t.t2a_first_perturbed_velocity, t.t2a_first_perturbed}, }; for (const Arm& arm : arms) { - INFO("arm = " << arm.name); + INFO("arm = " << std::string(arm.name)); REQUIRE(arm.velocity.size() == n); REQUIRE(arm.x0.size() == n); @@ -5596,3 +5886,794 @@ TEST_CASE("ltx2 t2a: rescale_scale 0 is the control because both spaces agree th CHECK(at_default > 1e-2); CHECK(at_default > 100.0 * at_zero); } + +// ─── row LTX25-GUIDED-VIDEO (#1092): the guided VIDEO denoiser ────────────── +// +// The video denoise loop ran ONE unguided forward per step and applied +// `ToDenoised` to it, while every recipe resolved a video guider that nothing +// read. These cases gate the four passes upstream's `_guided_denoise` assembles +// (ltx-pipelines utils/denoisers.py:97-137 @ fd4ded7f) and, for each of them, +// WHICH SPACE it was combined in. +// +// They enter through the production entry point — `LoadVideoEngine` then +// `VideoEngine::Generate`, which is what `vllm_video_generate` calls — on +// `pipeline_kind = one_stage`, whose OWN recipe resolves `cfg_scale = 3.0`, +// `stg_scale = 1.0`, `rescale_scale = 0.7` and `modality_scale = 3.0`. Nothing +// below constructs a guider, a DiT, a modality or a perturbation by hand. + +namespace { + +// `one_stage` on the shipped fixture, guided by its own recipe. The only guider +// field overridden is the STG block list, and `OneStageFixtureGuidance` says why. +vllm::multimodal::VideoModelParams OneStageParams(const ltx2_fixture::Paths& paths) { + vllm::multimodal::VideoModelParams mp = FixtureParams(paths); + mp.extras[vllm::multimodal::kLtx2PipelineKindExtra] = "one_stage"; + return mp; +} + +vllm::multimodal::VideoGenParams OneStageGen(const std::string& out_dir) { + vllm::multimodal::VideoGenParams gen = FixtureGen(out_dir); + gen.steps = 2; // two sigma intervals is enough to exercise the loop + OneStageFixtureGuidance(&gen); + return gen; +} + +double MaxAbsOf(const std::vector& v) { + double m = 0.0; + for (const float x : v) m = std::max(m, std::abs(static_cast(x))); + return m; +} + +double MaxAbsDiffOf(const std::vector& a, const std::vector& b) { + REQUIRE(a.size() == b.size()); + double m = 0.0; + for (size_t i = 0; i < a.size(); ++i) { + m = std::max(m, std::abs(static_cast(a[i]) - static_cast(b[i]))); + } + return m; +} + +} // namespace + +TEST_CASE("ltx2 one_stage: all four guidance arms are combined in X0 space (#1092)") { + // THE DEFECT THIS CASE EXISTS FOR, in two layers. + // + // The outer one is that the video path ran no guidance at all. That is caught + // by the pass counts below, which are read off the seam's own record of which + // forwards it issued rather than inferred from an output. + // + // The inner one is #1039's, on a path that never had it: `MultiModalGuider` + // combines DENOISED predictions, because `DiffusionStage` hands the loop an + // `X0Model` (utils/blocks.py:480-482) and not the raw velocity model. The + // guider's LINEAR terms are invariant under `x0 = latent - sigma*v`, so the + // difference is entirely in the rescale at guiders.py:268-271 — and + // `rescale_scale` is 0.7 on this recipe, which is the DEFAULT arm. + // + // WHAT MAKES THAT UNREACHABLE BY ACCIDENT. `cond == latent - sigma*velocity` is + // an EQUATION between three recorded tensors, not a magnitude. It is exact in + // x0 space; in velocity space `cond` IS the velocity and the residual is the + // whole sample. No fixture scale satisfies it by accident, a zeroed velocity + // collapses it to `cond == latent` and is refused by the lower bound below, + // and a zeroed `cond` fails it outright. + // + // ALL FOUR ARMS. #1039's first gate asserted the equation for the conditional + // pass alone; the T2A arm runs three forwards and three mutations survived + // that draft. This arm runs FOUR. + Workspace ws; + + // The arm this case sits on, pinned as a LOCAL fact before anything is read off + // a render. `rescale_scale = 0.7` on the 2.4/2.5 lineage (ltx-pipelines + // utils/constants.py:53 video / :63 audio, reached through `_PARAMS_SINCE_VERSION` at + // :130-133). + const vllm::Ltx2PipelineRecipe recipe = vllm::ResolveLtx2PipelineRecipe("one_stage", "2.5"); + REQUIRE(recipe.phases.size() == 1); + const vllm::Ltx2MultiModalGuiderParams row = recipe.phases[0].video_guidance; + CHECK(row.cfg_scale == 3.0); + CHECK(row.stg_scale == 1.0); + CHECK(row.rescale_scale == 0.7); + CHECK(row.modality_scale == 3.0); + + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(OneStageParams(ws.paths)); + REQUIRE(engine != nullptr); + (void)engine->Generate(OneStageGen(ws.root + "/guided_x0")); + const auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + const vllm::multimodal::Ltx2ConditioningTrace t = ltx->last_conditioning(); + REQUIRE(t.completed); + + // ── the render ran the guidance its recipe asked for ────────────────────── + // + // Counts, not tensors. An arm silently skipped changes a counter and changes no + // output shape, no frame count and no sample rate. + REQUIRE_MESSAGE(t.video_guided, "the video denoise did not go through the guided seam at all"); + CHECK(t.video_cond_forwards == 1); + CHECK_MESSAGE(t.video_uncond_forwards == 1, + "no unconditional forward ran, so `(cfg_scale - 1) * (cond - uncond)` is " + "identically zero on a recipe whose cfg_scale is 3.0"); + CHECK_MESSAGE(t.video_perturbed_forwards == 1, + "no perturbed forward ran, so `stg_scale * (cond - perturbed)` is identically " + "zero on a recipe whose stg_scale is 1.0"); + CHECK_MESSAGE(t.video_modality_forwards == 1, + "no isolated-modality forward ran, so `(modality_scale - 1) * (cond - mod)` is " + "identically zero on a recipe whose modality_scale is 3.0"); + // The guidance the ENGINE resolved, which is what the replay below uses. A + // build that resolved different scales fails the replay rather than agreeing + // with itself. + CHECK(t.video_guidance_cfg_scale == row.cfg_scale); + CHECK(t.video_guidance_stg_scale == row.stg_scale); + CHECK(t.video_guidance_rescale_scale == row.rescale_scale); + CHECK(t.video_guidance_modality_scale == row.modality_scale); + + // The perturbations REACHED the DiT, read off the mask that was handed over + // rather than off the guider params. A config that is BUILT and not HANDED + // OVER leaves the params untouched and renders. + CHECK(t.video_perturbed_blocks == std::vector{1}); + CHECK(t.video_audio_perturbed_blocks == std::vector{1}); + // WHAT THESE TWO MEASURE, said exactly, because the message they used to carry + // claimed more. `video_modality_skipped_{a2v,v2a}` is assigned from the + // `Ltx2DitPerturbation` THE SEAM BUILT and handed over + // (ltx2_denoisers.cpp:315-316), so it says the seam asked for both directions + // — which is `blocks=None` on both types (denoisers.py:125-138) — and says + // NOTHING about what the DiT did with the request. What the DiT does with each + // flag is gated separately and per direction by + // "ltx2 dit: each CROSS perturbation gates ITS OWN direction and no other", + // which exists because mutations that applied exactly one direction, or + // swapped the two, survived this case. + CHECK_MESSAGE(t.video_modality_skipped_a2v, + "the seam built the isolated-modality pass WITHOUT asking for SKIP_A2V_CROSS_ATTN, " + "so that pass is the conditional pass again in the audio->video direction " + "(denoisers.py:125-138)"); + CHECK_MESSAGE(t.video_modality_skipped_v2a, + "the seam built the isolated-modality pass WITHOUT asking for " + "SKIP_V2A_CROSS_ATTN"); + + const size_t n = t.video_first_latent.size(); + REQUIRE(n > 0); + const size_t tokens = t.video_first_timesteps.size(); + REQUIRE(tokens > 0); + const size_t width = n / tokens; + REQUIRE(width * tokens == n); + + // THE FIXTURE CAN DECIDE THIS AT ALL. The two candidate tensors for every arm + // are `latent - sigma*velocity` and `velocity`, and they coincide when the + // sample is zero. A REQUIRE, because nothing below discriminates once it fails. + const double latent_span = MaxAbsOf(t.video_first_latent); + REQUIRE_MESSAGE(latent_span > 1e-3, + "the step-0 sample is zero, so the two candidate tensors coincide and nothing " + "below discriminates"); + + // ── the equation, once per guidance pass ────────────────────────────────── + struct Arm { + const char* name; + const std::vector& velocity; + const std::vector& x0; + }; + const Arm arms[] = { + {"cond", t.video_first_cond_velocity, t.video_first_cond}, + {"uncond", t.video_first_uncond_velocity, t.video_first_uncond}, + {"perturbed", t.video_first_perturbed_velocity, t.video_first_perturbed}, + {"modality", t.video_first_modality_velocity, t.video_first_modality}, + }; + for (const Arm& arm : arms) { + INFO("arm = " << std::string(arm.name)); + REQUIRE(arm.velocity.size() == n); + REQUIRE(arm.x0.size() == n); + + double velocity_span = 0.0; + double sigma_velocity_span = 0.0; + double err_x0 = 0.0; // |x0 - (latent - sigma*velocity)| -> 0 in x0 space + double err_v = 0.0; // |x0 - velocity| -> 0 in velocity space + for (size_t token = 0; token < tokens; ++token) { + // The PER-TOKEN sigma, which is what `X0Model.forward` uses + // (model.py:601-604 passes `video.timesteps`). Using the schedule scalar + // here would pass on a build that used it too, and that build re-noises + // every conditioned token. + const double sigma = static_cast(t.video_first_timesteps[token]); + for (size_t c = 0; c < width; ++c) { + const size_t i = token * width + c; + const double lat = static_cast(t.video_first_latent[i]); + const double vel = static_cast(arm.velocity[i]); + const double x0 = static_cast(arm.x0[i]); + velocity_span = std::max(velocity_span, std::abs(vel)); + sigma_velocity_span = std::max(sigma_velocity_span, std::abs(sigma * vel)); + err_x0 = std::max(err_x0, std::abs(x0 - (lat - sigma * vel))); + err_v = std::max(err_v, std::abs(x0 - vel)); + } + } + INFO("max|latent| = " << latent_span << " max|velocity| = " << velocity_span + << " max|sigma*velocity| = " << sigma_velocity_span + << " |x0 - (latent - sigma*velocity)| = " << err_x0 + << " |x0 - velocity| = " << err_v << " elements = " << n); + + // 1. `to_denoised` IS NOT THE IDENTITY ON THIS ARM. The second half of the + // non-vacuity guard, per arm: a zeroed velocity collapses the equation to + // `x0 == latent` and would let a stub satisfy it. + REQUIRE_MESSAGE(sigma_velocity_span > 1e-6, + "the DiT returned no velocity on this arm, so `to_denoised` is the identity " + "here and the two candidate tensors coincide"); + // 2. THE GUIDER WAS HANDED THE X0 PREDICTION, exactly. + CHECK_MESSAGE(err_x0 <= 1e-5 * latent_span, + "the tensor handed to `Ltx2MultiModalGuidance` on this arm is not " + "`latent - sigma*velocity`, which is what `X0Model.forward` returns " + "(model.py:590-604, #1039): residual " + << err_x0 << " against a tolerance of " << (1e-5 * latent_span)); + // 3. AND IT WAS NOT THE RAW VELOCITY. Said separately from check 2, because a + // build handing the guider some THIRD tensor fails 2 and would pass a lone + // "not the velocity" check; the pair says which of the two happened. + CHECK_MESSAGE(err_v > 1e-2 * latent_span, + "the tensor handed to `Ltx2MultiModalGuidance` on this arm IS the raw DiT " + "velocity, so the guidance is combined in velocity space and converted once " + "afterwards (#1039)"); + } + + // ── each arm is a DIFFERENT forward ─────────────────────────────────────── + // + // Without these, an arm whose CONTEXT or PERTURBATION never reached the DiT + // satisfies every check above: it is a perfectly converted x0 prediction of the + // conditional pass, and its guidance term is exactly zero. + CHECK_MESSAGE(MaxAbsDiffOf(t.video_first_uncond, t.video_first_cond) > 1e-6 * latent_span, + "the unconditional pass returned the conditional pass's own tensor, so the " + "negative context did not reach the forward"); + CHECK_MESSAGE(MaxAbsDiffOf(t.video_first_perturbed, t.video_first_cond) > 1e-6 * latent_span, + "the perturbed pass returned the conditional pass's own tensor, so the " + "self-attention perturbation did not reach the forward"); + CHECK_MESSAGE(MaxAbsDiffOf(t.video_first_modality, t.video_first_cond) > 1e-6 * latent_span, + "the isolated-modality pass returned the conditional pass's own tensor, so the " + "cross-attention perturbation did not reach the forward (transformer.py:335,367)"); + + // ── the guider's output is the guider's output ──────────────────────────── + // + // The four recorded arms, through the SHIPPED `Ltx2MultiModalGuidance` on the + // recipe's own params, must reproduce `video_first_denoised` bit for bit. This + // does not gate the guider's arithmetic — the seam case below does that — it + // gates that the pipeline handed the guider these tensors and passed its result + // on UNTOUCHED. A second `to_denoised` applied to the combination is invisible + // in every per-arm check above, because it moves nothing the guider was handed. + { + const std::vector replayed = vllm::Ltx2MultiModalGuidance( + row, t.video_first_cond.data(), t.video_first_uncond.data(), + t.video_first_perturbed.data(), t.video_first_modality.data(), static_cast(n)); + REQUIRE(replayed.size() == n); + const double worst = MaxAbsDiffOf(replayed, t.video_first_denoised); + INFO("max|replayed guidance - video_first_denoised| = " << worst); + // EXACT, not a tolerance: it is the same function over the same f32 inputs, + // so any non-zero residual is another operation this pipeline applied. + CHECK_MESSAGE(worst == 0.0, + "`video_first_denoised` is not `Ltx2MultiModalGuidance` over the four recorded " + "arms, so something else was applied to the guider's result (#1039)"); + // And the combination MOVED what it was handed, so the arms checked above are + // real inputs to it rather than recorded values beside one. + CHECK(t.video_first_denoised != t.video_first_cond); + } + + // ── the same combination, over arms REBUILT FROM THE RAW VELOCITIES ─────── + // + // WHY THIS IS NOT THE PREVIOUS CHECK AGAIN. The replay above is fed the arms + // the seam recorded, so anything applied to EVERY arm on the way out of the + // forward is invisible to it: the replay and the pipeline agree because they + // agree about the same altered inputs. `post_process_latent` applied per arm + // instead of once to the guider's result is exactly that shape, and it is not + // hypothetical -- upstream applies it in the LOOP (utils/samplers.py:35), one + // level above the denoiser, and applying it a level lower is the obvious + // simplification. + // + // It is also invisible to the per-arm invariant, and that took working out. + // `post_process_latent` is `x*mask + clean*(1-mask)`, so it only moves tokens + // whose denoise mask is 0 -- and on such a token the schedule sigma is 0 too + // (`timesteps_from_mask`, utils/helpers.py:494-503), so the invariant reads + // `x0 == latent`, and a conditioned token's `latent` IS its clean value. The + // two placements therefore agree token by token and differ only through + // `cond.std()` and `pred.std()`, which the rescale computes over the WHOLE + // tensor and which change for every element at once. + // + // Rebuilding the arms from `latent` and the raw velocities is independent of + // anything applied to the arms, so it sees that. It is exact rather than + // approximate because it repeats `ToDenoised`'s own arithmetic: the subtraction + // in double, the store in f32. + { + const auto rebuild = [&](const std::vector& velocity) { + std::vector out(n); + for (size_t token = 0; token < tokens; ++token) { + const double sigma = static_cast(t.video_first_timesteps[token]); + for (size_t c = 0; c < width; ++c) { + const size_t i = token * width + c; + out[i] = static_cast(static_cast(t.video_first_latent[i]) - + sigma * static_cast(velocity[i])); + } + } + return out; + }; + const std::vector c = rebuild(t.video_first_cond_velocity); + const std::vector u = rebuild(t.video_first_uncond_velocity); + const std::vector p = rebuild(t.video_first_perturbed_velocity); + const std::vector m = rebuild(t.video_first_modality_velocity); + const std::vector replayed = vllm::Ltx2MultiModalGuidance( + row, c.data(), u.data(), p.data(), m.data(), static_cast(n)); + const double worst = MaxAbsDiffOf(replayed, t.video_first_denoised); + INFO("max|guidance over rebuilt arms - video_first_denoised| = " << worst); + CHECK_MESSAGE(worst == 0.0, + "the guider's result is not `Ltx2MultiModalGuidance` over `latent - sigma*v` " + "for the four RAW velocities, so something was applied to the arms between the " + "forward and the combination"); + } + + // ── `post_process_latent` came AFTER the guider, and the sampler consumed + // exactly what it produced ───────────────────────────────────────────── + // + // `_step_state` applies `post_process_latent(denoised, ...)` to the DENOISER's + // result (utils/samplers.py:35), never per arm inside it. On this render no + // token is conditioned, so the two tensors coincide — asserted rather than + // assumed, because it is what makes the Euler recovery below a statement about + // `video_first_denoised`. + REQUIRE(t.video_first_stepper_input.size() == n); + CHECK(t.video_first_stepper_input == t.video_first_denoised); + + { + // `Ltx2EulerStep` is `x + (x - denoised)/sigma * (sigma_next - sigma)` + // (`EulerDiffusionStep`, ltx-pipelines utils/blocks.py:524-527). Recovering + // `video_first_next_latent` from `video_first_stepper_input` pins WHICH + // tensor the step was handed: `ToDenoised` applied a second time between the + // recording and the step leaves every field above untouched and moves only + // this one. + // + // The schedule is re-derived from the shared seam rather than read off the + // render, and tied to it by the sigma the render recorded. + REQUIRE(t.schedule_tokens > 0); + const std::vector sigmas = vllm::Ltx2SigmaSchedule(/*steps=*/2, t.schedule_tokens); + REQUIRE(sigmas.size() == 3); + const double sigma = t.video_first_sigma; + REQUIRE(sigma > 0.0); + REQUIRE(static_cast(sigmas[0]) == sigma); + const double dt = static_cast(sigmas[1]) - static_cast(sigmas[0]); + REQUIRE_MESSAGE(std::abs(dt) > 1e-3, + "the first two sigmas coincide, so the Euler step is the identity and this " + "check cannot see what it consumed"); + REQUIRE(t.video_first_next_latent.size() == n); + double worst = 0.0; + double scale = 0.0; + for (size_t i = 0; i < n; ++i) { + const double lat = static_cast(t.video_first_latent[i]); + const double den = static_cast(t.video_first_stepper_input[i]); + const double expected = lat + (lat - den) / sigma * dt; + worst = + std::max(worst, std::abs(static_cast(t.video_first_next_latent[i]) - expected)); + scale = std::max(scale, std::abs(expected)); + } + INFO("sigma = " << sigma << " -> " << sigmas[1] + << " max|next - Euler(latent, denoised)| = " << worst + << " scale = " << scale); + REQUIRE_MESSAGE(scale > 1e-3, + "the recomputed Euler output is zero, so the residual below bounds nothing"); + CHECK_MESSAGE(worst <= 1e-5 * scale, + "the latent `Ltx2EulerStep` wrote is not the step over the recorded denoised " + "prediction, so the sampler was handed some other tensor (#1039): residual " + << worst << " against a tolerance of " << (1e-5 * scale)); + } +} + +TEST_CASE("ltx2 one_stage: rescale_scale 0 is the control and the modality term is INERT in it") { + // #1039's control on the VIDEO row. It runs with `modality_scale = 3.0`, which + // the T2A control could not carry because that pipeline pins it to 1.0 + // (t2a_one_stage.py:202) — so the isolated-modality arm is inside a space + // control here for the first time. + // + // WHAT THAT IS WORTH, measured rather than implied, and the case's own title + // said more than the number supports until 2026-08-17. Presence is coverage, + // not discriminating power: the third measurement below pins `modality_scale` + // to 1.0 and the disagreement at the shipped rescale barely moves. A reader + // must not lean on this control for modality coverage. THE MODALITY ARM'S GATE + // IS THE PER-ARM INVARIANT in the case above, whose `modality` row is the one + // mutation M4 (the `mod` pass left in velocity space) turns red; this control + // gates the RESCALE, on a guider that happens to have four terms. + // + // The case above would be testing something OTHER than the defect if it also + // fired at `rescale_scale = 0`, because `MultiModalGuider.calculate`'s linear + // terms (guiders.py:261-266) are invariant under `x0 = latent - sigma*v`: + // + // latent - sigma*(c + a(c-u) + b(c-p) + d(c-m)) + // == x0c + a(x0c-x0u) + b(x0c-x0p) + d(x0c-x0m) + // + // The rescale at `:268-271` is the only part that is not. + const int64_t n = 512; + std::vector latent(static_cast(n)); + std::vector v_cond(static_cast(n)); + std::vector v_uncond(static_cast(n)); + std::vector v_ptb(static_cast(n)); + std::vector v_mod(static_cast(n)); + // Deterministic and NON-CONSTANT. A zero latent erases `(factor - 1) * latent` + // entirely and a constant one reduces it to a uniform offset; either would make + // the disagreement below unmeasurable and the control meaningless. + uint64_t s = 0x9E3779B97F4A7C15ULL; + const auto next = [&s]() { + s ^= s << 13; + s ^= s >> 7; + s ^= s << 17; + return static_cast(static_cast(s >> 11) / 9007199254740992.0 * 2.0 - 1.0); + }; + for (int64_t i = 0; i < n; ++i) { + const size_t j = static_cast(i); + latent[j] = 2.0F * next(); + v_cond[j] = next(); + v_uncond[j] = next(); + v_ptb[j] = next(); + v_mod[j] = next(); + } + const float sigma = 0.83F; + std::vector x_cond(static_cast(n)); + std::vector x_uncond(static_cast(n)); + std::vector x_ptb(static_cast(n)); + std::vector x_mod(static_cast(n)); + for (int64_t i = 0; i < n; ++i) { + const size_t j = static_cast(i); + x_cond[j] = latent[j] - sigma * v_cond[j]; + x_uncond[j] = latent[j] - sigma * v_uncond[j]; + x_ptb[j] = latent[j] - sigma * v_ptb[j]; + x_mod[j] = latent[j] - sigma * v_mod[j]; + } + + // The 2.4/2.5 VIDEO row, read from the shared recipe table rather than typed. + const vllm::Ltx2PipelineRecipe recipe = vllm::ResolveLtx2PipelineRecipe("one_stage", "2.5"); + REQUIRE(recipe.phases.size() == 1); + vllm::Ltx2MultiModalGuiderParams params = recipe.phases[0].video_guidance; + REQUIRE(params.rescale_scale == 0.7); + REQUIRE(params.modality_scale == 3.0); + + const auto compare = [&](double rescale) { + params.rescale_scale = rescale; + // Upstream's shape: combine the X0 predictions. + const std::vector x0_space = vllm::Ltx2MultiModalGuidance( + params, x_cond.data(), x_uncond.data(), x_ptb.data(), x_mod.data(), n); + // The shape a port reaches for by accident: combine the VELOCITIES and + // convert once after. + const std::vector v_space = vllm::Ltx2MultiModalGuidance( + params, v_cond.data(), v_uncond.data(), v_ptb.data(), v_mod.data(), n); + double worst = 0.0; + double scale = 0.0; + for (int64_t i = 0; i < n; ++i) { + const size_t j = static_cast(i); + const double converted = static_cast(latent[j]) - + static_cast(sigma) * static_cast(v_space[j]); + worst = std::max(worst, std::abs(static_cast(x0_space[j]) - converted)); + scale = std::max(scale, std::abs(static_cast(x0_space[j]))); + } + REQUIRE(scale > 1e-3); + return worst / scale; + }; + + const double at_zero = compare(0.0); + const double at_default = compare(0.7); + // The same pair with the modality term switched OFF, which is what T2A's + // control already measured. Restored afterwards so nothing below reads a + // mutated params object. + const double shipped_modality = params.modality_scale; + params.modality_scale = 1.0; + const double at_zero_no_modality = compare(0.0); + const double at_default_no_modality = compare(0.7); + params.modality_scale = shipped_modality; + + INFO("relative disagreement: at rescale 0.0 = " + << at_zero << " at rescale 0.7 = " << at_default + << " | modality_scale pinned to 1.0: at 0.0 = " << at_zero_no_modality + << " at 0.7 = " << at_default_no_modality); + // AT 0.0 THE TWO SPACES ARE THE SAME FUNCTION, to f32 rounding — with the + // modality term present, which is the arm this control adds over T2A's. + CHECK(at_zero < 1e-4); + // AT THE SHIPPED 0.7 THEY ARE NOT, by orders of magnitude more. + CHECK(at_default > 1e-2); + CHECK(at_default > 100.0 * at_zero); + + // AND THE MODALITY TERM IS NOT WHAT SEPARATES THEM. Asserted rather than left + // in prose, because the case's own comment implied the opposite and a later + // reader would otherwise treat this control as modality coverage. The two + // `0.7` numbers agree to well inside a factor of two: adding a fourth linear + // term changes what the rescale is computed over and does not change whether + // the rescale is the term that breaks the equivalence. + CHECK(at_zero_no_modality < 1e-4); + CHECK(at_default_no_modality > 1e-2); + CHECK_MESSAGE(at_default_no_modality > 0.5 * at_default, + "the modality term turned out to carry the disagreement after all, which would " + "make this control modality coverage rather than rescale coverage"); + CHECK_MESSAGE(at_default_no_modality < 2.0 * at_default, + "the modality term turned out to carry the disagreement after all"); +} + +TEST_CASE("ltx2 one_stage: post_process_latent runs AFTER the guider, not per arm (#1092)") { + // WHERE `post_process_latent` IS APPLIED, gated on a render that has something + // for it to move. The unconditioned case above cannot see this at all: every + // denoise mask entry is 1 there, so `x*mask + clean*(1-mask)` + // (utils/helpers.py:462-464) is a literal no-op and any placement of it passes. + // + // TWO THINGS ARE TRUE HERE AND THEY ARE EASY TO CONFUSE, so both are asserted. + // + // (1) Applying it to each ARM is an IDENTITY, and that is not a gap in this + // case -- it is arithmetic. A conditioned token arrives with its per-token + // sigma at 0 (`timesteps_from_mask`, utils/helpers.py:494-503), so + // `X0Model` returns `latent - 0*v`, which is `latent`; and a conditioned + // token's `latent` IS its clean value, which is what the conditioner wrote + // and what the Euler step preserves. So every arm already equals what + // post-processing would write. MEASURED: adding it per arm runs the whole + // suite to 71 cases / 2145 assertions / exit 0, and the arm assertion below + // is what says WHY rather than leaving the green unexplained. + // + // (2) Applying it after the GUIDER is emphatically not an identity, and that is + // the thing worth gating. The guider's rescale (guiders.py:268-271) is a + // scalar over the WHOLE tensor, so it multiplies the conditioned tokens too + // -- `pred = latent * factor` there, because every guidance term is zero on + // a token where all four arms agree. `post_process_latent` is what pins + // them back to `clean`. Take it out, or move it a level down into the + // denoiser, and the conditioned tokens leave the step scaled by a number + // nobody asked for, on a render that finishes. + // + // So this case asserts that the arms were NOT touched and that the guider's + // result WAS, on exactly the mask-0 tokens. + Workspace ws; + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(OneStageParams(ws.paths)); + REQUIRE(engine != nullptr); + vllm::multimodal::VideoGenParams gen = OneStageGen(ws.root + "/conditioned"); + gen.first_frame_ppm = ConditioningPpm(20, 28, 1); + gen.extras[vllm::multimodal::kLtx2ImageCrfExtra] = "0"; + (void)engine->Generate(gen); + const auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + const vllm::multimodal::Ltx2ConditioningTrace t = ltx->last_conditioning(); + REQUIRE(t.completed); + REQUIRE(t.video_guided); + // The same four arms as the case above, not a degenerate set. + CHECK(t.video_uncond_forwards == 1); + CHECK(t.video_perturbed_forwards == 1); + CHECK(t.video_modality_forwards == 1); + + const size_t tokens = t.video_first_timesteps.size(); + REQUIRE(tokens > 0); + const size_t n = t.video_first_latent.size(); + REQUIRE(n > 0); + const size_t width = n / tokens; + REQUIRE(width * tokens == n); + + // NON-VACUITY, both ends. With no conditioned token this case is the one above + // again; with every token conditioned there is nothing left to denoise. + size_t conditioned = 0; + for (size_t token = 0; token < tokens; ++token) { + if (t.video_first_timesteps[token] == 0.0F) ++conditioned; + } + INFO("conditioned tokens = " << conditioned << " of " << tokens); + REQUIRE_MESSAGE(conditioned > 0, + "no token arrived at the denoiser with a zero timestep, so the image " + "conditioning did not reach the denoise mask and this case tests nothing"); + REQUIRE_MESSAGE(conditioned < tokens, + "EVERY token is conditioned, so there is nothing left to denoise"); + + // (2), AND THE POSITIVE CONTROL FOR THE WHOLE CASE. `post_process_latent` MOVES + // something on this render: the tensor the stepper was handed is not the + // guider's own output. Without this, every assertion here would be satisfied by + // a render where post-processing happened to be a no-op, which is exactly what + // the unconditioned case above is. + REQUIRE(t.video_first_stepper_input.size() == n); + REQUIRE_MESSAGE(t.video_first_stepper_input != t.video_first_denoised, + "`post_process_latent` changed nothing on this render, so it cannot matter " + "WHERE it was applied and this case discriminates nothing"); + + // AND IT MOVED ONLY THE CONDITIONED TOKENS, which is what makes the next + // assertion a statement about placement rather than about some third operation. + for (size_t token = 0; token < tokens; ++token) { + const bool is_conditioned = t.video_first_timesteps[token] == 0.0F; + for (size_t c = 0; c < width; ++c) { + const size_t i = token * width + c; + const bool moved = t.video_first_stepper_input[i] != t.video_first_denoised[i]; + if (moved == is_conditioned) continue; + INFO("token = " << token << " channel = " << c); + FAIL_CHECK("`post_process_latent` moved a token whose denoise mask does not match: it is " + "`x*mask + clean*(1-mask)` and must move exactly the mask-0 tokens"); + break; + } + } + + // (1). Every arm the forward returned is `latent - sigma*velocity`, INCLUDING + // on the conditioned tokens, where that is `latent` itself. This is what makes + // the per-arm placement an identity rather than an undetected defect, and it is + // asserted rather than argued because the argument depends on a conditioned + // token's `latent` being its clean value -- a property of the CONDITIONER, one + // file away, that nothing here would otherwise hold. + const std::vector* arms[] = {&t.video_first_cond, &t.video_first_uncond, + &t.video_first_perturbed, &t.video_first_modality}; + const std::vector* velocities[] = { + &t.video_first_cond_velocity, &t.video_first_uncond_velocity, + &t.video_first_perturbed_velocity, &t.video_first_modality_velocity}; + const char* names[] = {"cond", "uncond", "perturbed", "modality"}; + for (size_t k = 0; k < 4; ++k) { + INFO("arm = " << std::string(names[k])); + REQUIRE(arms[k]->size() == n); + REQUIRE(velocities[k]->size() == n); + double worst = 0.0; + for (size_t token = 0; token < tokens; ++token) { + const double sigma = static_cast(t.video_first_timesteps[token]); + for (size_t c = 0; c < width; ++c) { + const size_t i = token * width + c; + // `ToDenoised` subtracts in double and STORES f32, so the expectation is + // rounded the same way. Comparing against the unrounded double leaves one + // ULP of disagreement -- measured at 5.96e-08, which is 2^-24 -- and a + // tolerance wide enough to absorb it would also absorb a real defect an + // order of magnitude away. + const float expected = static_cast(static_cast(t.video_first_latent[i]) - + sigma * static_cast((*velocities[k])[i])); + worst = std::max(worst, std::abs(static_cast((*arms[k])[i]) - + static_cast(expected))); + } + } + INFO("max|arm - (latent - sigma*velocity)| = " << worst); + CHECK_MESSAGE(worst == 0.0, + "this arm is not `latent - sigma*velocity` on every token, so something was " + "applied to it between the forward and the guider -- and if that something is " + "`post_process_latent`, it has stopped being an identity on the arms and the " + "per-arm placement is now a real divergence rather than a harmless one"); + } +} + +TEST_CASE("ltx2 guided video: the refusals that would otherwise RENDER (#1092)") { + Workspace ws; + + SUBCASE("an unconditional forward with no negative conditioning is refused BY NAME") { + // The positive embeds alone, which is what every engine here loaded before + // this row. `cfg_scale = 3.0` asks for a forward whose context does not + // exist; serving the POSITIVE context twice would make the whole CFG term + // exactly zero and render an unguided clip wearing a guided configuration. + vllm::multimodal::VideoModelParams mp = OneStageParams(ws.paths); + mp.extras.erase(vllm::multimodal::kLtx2NegativePromptEmbedsExtra); + mp.extras.erase(vllm::multimodal::kLtx2NegativeAudioPromptEmbedsExtra); + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + try { + (void)engine->Generate(OneStageGen(ws.root + "/no_negative")); + FAIL("a cfg_scale of 3.0 with no negative conditioning must be refused"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find("unconditional forward") != std::string::npos); + CHECK(msg.find(vllm::multimodal::kLtx2NegativePromptEmbedsExtra) != std::string::npos); + } + } + + SUBCASE("cfg_scale 1.0 turns the unconditional pass off instead of needing one") { + // The other half of the branch above, and what makes it a statement about the + // GUIDER rather than a blanket requirement: `do_unconditional_generation` is + // `not isclose(cfg_scale, 1.0)` (guiders.py:275-277), so at 1.0 there is no + // pass and nothing to encode. + vllm::multimodal::VideoModelParams mp = OneStageParams(ws.paths); + mp.extras.erase(vllm::multimodal::kLtx2NegativePromptEmbedsExtra); + mp.extras.erase(vllm::multimodal::kLtx2NegativeAudioPromptEmbedsExtra); + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + vllm::multimodal::VideoGenParams gen = OneStageGen(ws.root + "/cfg_one"); + gen.extras[vllm::multimodal::kLtx2VideoCfgScaleExtra] = "1.0"; + gen.extras[vllm::multimodal::kLtx2AudioCfgScaleExtra] = "1.0"; + const vllm::multimodal::VideoResult result = engine->Generate(gen); + CHECK(result.width == 64); + const auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + const vllm::multimodal::Ltx2ConditioningTrace t = ltx->last_conditioning(); + CHECK(t.video_uncond_forwards == 0); + // And the OTHER two passes still ran, so this subcase turned off exactly one + // arm rather than the guidance. + CHECK(t.video_perturbed_forwards == 1); + CHECK(t.video_modality_forwards == 1); + } + + SUBCASE("an STG block this checkpoint does not have is refused, not silently ignored") { + // `Perturbation.is_perturbed` is `block in self.blocks` + // (guidance/perturbations.py:26-33), so a block index past the end perturbs + // NOTHING: the perturbed forward returns the conditional pass's own tensor + // and `stg_scale * (cond - perturbed)` is exactly zero. The render is finite, + // the right size, and carries no spatio-temporal guidance whatever. + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(OneStageParams(ws.paths)); + vllm::multimodal::VideoGenParams gen = OneStageGen(ws.root + "/stg_oob"); + gen.extras[vllm::multimodal::kLtx2VideoStgBlocksExtra] = "28"; + try { + (void)engine->Generate(gen); + FAIL("block 28 on a two-block DiT perturbs nothing and must be refused"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find("stg_blocks") != std::string::npos); + CHECK(msg.find("exactly zero") != std::string::npos); + } + } + + SUBCASE("an EMPTY stg_blocks is SERVED - it is upstream's own way to disable STG") { + // THIS SUBCASE ASSERTED A REFUSAL UNTIL 2026-08-17. Measured at + // Lightricks/LTX-2 `fd4ded7f`: `docs/multimodal-guidance.md:13` documents + // "Set to `[]` to disable STG"; `MultiModalGuiderParams.stg_blocks` defaults + // to `[]` (guiders.py:204); the flags are `nargs="*"` (args.py:979-985) so + // the empty list has a CLI spelling; `LTX_2_3_HQ_PARAMS` ships it on both + // modalities (constants.py:105, :113); and nothing in that tree validates + // the list at all. Refusing it made this port reject a configuration its + // reference documents, ships and cannot express any other way. + // + // Upstream does NOT skip the pass either: `do_perturbed_generation` reads + // `stg_scale` alone (guiders.py:279-281), so the "ptb" entry is appended and + // the batch carries a sample whose result equals `cond`. The forward count + // below is that fact, and it is why an empty list disables the STG SIGNAL + // and not the STG COST. + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(OneStageParams(ws.paths)); + REQUIRE(engine != nullptr); + vllm::multimodal::VideoGenParams gen = OneStageGen(ws.root + "/stg_empty"); + gen.extras[vllm::multimodal::kLtx2VideoStgBlocksExtra] = ""; + gen.extras[vllm::multimodal::kLtx2AudioStgBlocksExtra] = ""; + (void)engine->Generate(gen); // it RENDERS; a throw fails the case + const auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + const vllm::multimodal::Ltx2ConditioningTrace t = ltx->last_conditioning(); + REQUIRE(t.completed); + REQUIRE(t.video_guided); + // The pass still ran, because the scale still asks for it. + CHECK(t.video_perturbed_forwards == 1); + // And it perturbed nothing, read off the mask handed to the DiT. + CHECK(t.video_perturbed_blocks.empty()); + CHECK(t.video_audio_perturbed_blocks.empty()); + // So the STG term is not merely small, it is EXACTLY zero: the perturbed arm + // is the conditional arm bit for bit. An exact comparison, because a + // tolerance here would also pass on a build that perturbed a block and + // happened to move little. + REQUIRE(!t.video_first_cond.empty()); + CHECK_MESSAGE(t.video_first_perturbed == t.video_first_cond, + "an empty stg_blocks perturbed something, so PRESENT-and-empty was collapsed " + "onto some other value (`blocks=None` is EVERY block upstream, " + "perturbations.py:26-33)"); + // The control that this is about EMPTINESS and not about the extra being + // read at all: the same render with a real block moves the arm. + vllm::multimodal::VideoGenParams named = OneStageGen(ws.root + "/stg_named"); + named.extras[vllm::multimodal::kLtx2VideoStgBlocksExtra] = "1"; + named.extras[vllm::multimodal::kLtx2AudioStgBlocksExtra] = "1"; + (void)engine->Generate(named); + const vllm::multimodal::Ltx2ConditioningTrace n = ltx->last_conditioning(); + CHECK(n.video_perturbed_blocks == std::vector{1}); + CHECK(n.video_first_perturbed != n.video_first_cond); + } + + SUBCASE("a recipe that fixes its guidance refuses the override rather than applying it") { + // `allow_guidance_override = false` on the distilled two-stage recipe + // (ltx2_recipes.py:125-158), whose scales are distilled INTO the weights. + // Until this row nothing read that field at all. + vllm::multimodal::VideoModelParams mp = FixtureParams(ws.paths); + mp.extras["upsampler_path"] = ws.paths.upsampler; // the two-stage recipe's phase 1 + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + vllm::multimodal::VideoGenParams gen = FixtureGen(ws.root + "/fixed_guidance"); + gen.extras[vllm::multimodal::kLtx2VideoCfgScaleExtra] = "5.0"; + try { + (void)engine->Generate(gen); + FAIL("the distilled recipe fixes its guidance and must refuse the override"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find("fixes its own guidance") != std::string::npos); + CHECK(msg.find(vllm::multimodal::kLtx2VideoCfgScaleExtra) != std::string::npos); + } + } + + SUBCASE("the DISTILLED recipe runs ONE forward, which is what SimpleDenoiser is") { + // The guided seam is on every video render now, so the recipes upstream + // denoises with `SimpleDenoiser` (distilled.py:266,295) must still issue one + // forward per step. Their guiders are `Ltx2MultiModalGuiderParams`'s own + // defaults, which is `_POSITIVE_ONLY_GUIDER` (denoisers.py:25-28). + vllm::multimodal::VideoModelParams mp = FixtureParams(ws.paths); + mp.extras["upsampler_path"] = ws.paths.upsampler; // the two-stage recipe's phase 1 + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + (void)engine->Generate(FixtureGen(ws.root + "/distilled_simple")); + const auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + const vllm::multimodal::Ltx2ConditioningTrace t = ltx->last_conditioning(); + REQUIRE(t.video_guided); + CHECK(t.video_cond_forwards == 1); + CHECK(t.video_uncond_forwards == 0); + CHECK(t.video_perturbed_forwards == 0); + CHECK(t.video_modality_forwards == 0); + // And the guider was the identity over that one pass, so this recipe's + // trajectory is unchanged by the seam. + CHECK(t.video_first_denoised == t.video_first_cond); + } +}