feat(LTX25-TOKEN-APPEND): grow the phase loop's token sequence, and trim it back (#930) - #964
Merged
Merged
Conversation
…rms (#930) Two rows hit the same wall hours apart without talking to each other. #930 was filed from the IC-LoRA port and #920 from the generated-keyframe-slots port, and both stopped at the same sentence: this engine's phase loop is fixed at one `Ltx2VideoTokenCount(vshape, 1)`. A limitation two independent rows reach by different routes is a shared seam, so it gets its own spec. The sizing in #930 is corrected here rather than inherited. It names two blocked arms because it predates #920; there are three — reference video, the last-frame keyframe, and generated keyframe slots. Three findings the spec records before any code, because each one moves what the row is allowed to build: The appending conditioning ITEMS are already ported and gated. What is missing is `extend_keyframes_mask` (mask_utils.py:74-105), which upstream's docstring makes an obligation of appending itself, and `clear_conditioning` (tools.py:88-117), which trims back to the target count and restores an ALL-ONES mask rather than slicing the conditioned one. The attention mask is NOT the gap, so no field is added for it. Both ported video items pass a literal `attention_mask=None` (keyframe_cond.py:68-76, reference_video_cond.py:88-96) and the only route to a non-None mask is `ConditioningItemAttentionStrengthWrapper`, applied solely at iclora_utils.py:169 on the IC-LoRA path. A field here would be one no ported item could populate. The sigma schedule must keep reading the TARGET count. Upstream derives its shift from `math.prod(latent.shape[2:])` (schedulers.py:32), the unpatchified target, and the pipelines compute sigmas before the state exists at all. This engine's call sits AFTER the conditioning block, so it re-shifts the whole schedule the moment anything appends. Scope is the seam plus one arm lifted as its demonstration. Reference video stays refused and its message stays byte-identical: at this base PR #938 is open and unmerged, so the IC-LoRA metadata cause that refusal names is still true, and rewording it onto token-append alone would ship a refusal naming the first of two causes. The spec also records, up front, three limits a reviewer should press on rather than rediscover, all of them found while running the mutation pass rather than reasoned about afterwards. The sigma-schedule check has a residual, because `schedule_tokens` is one local feeding both the schedule and the trace, so the ordinary mutation moves both and REDs while a mutation editing only the call argument would not be caught; nothing local can close that, since the instrument and the measured expression would have to become the same thing. The trim is gated on a guard rather than on pixels, because appended tokens sit at the tail of a contiguous buffer and `Ltx2VideoUnpatchify` takes a bare pointer, so an un-trimmed state renders pixel-identical frames. And `Ltx2ExtendKeyframesMask(marked=true)` lands with a unit driver and no product caller; it is under `## Owed`, owned by row `LTX25-GENERATED-KEYFRAMES` and tracked by #920. The upstream anchors here are the corrected ones. `tools.py:103` is `clean_latent = ...[:, :num_tokens]`; the all-ones restore is :104. `tools.py:112` is `attention_mask=None`; the keyframes-mask drop is :113. And `schedulers.py:38-39` is the shift arithmetic, not the `tokens = math.prod(latent.shape[2:])` it was quoted for, which is :32. An anchor that resolves to a real but different statement is worse than a dangling one: a reader who checks is misled rather than alerted. The issue-index row lands SHA-anchored in the same commit, because that file is append-only and the row cannot be corrected after landing. WHY THIS SPEC LANDS TWICE IN `git log`. It was first committed on `row/LTX25-TOKEN-APPEND` (PR #948, head a1b1d8d) and reviewed there. That branch cannot land, because `scripts/check-doc-checkpoint.py` walks a range PER COMMIT (#573) and the review-repair commit b5618b3 edited `include/vllm/` headers — `user_usage` — while the `docs/USAGE.md` edit that pays for them sat in a sibling commit. The final tree satisfied the obligation; only the ordering failed. A commit message cannot be corrected in place and this repository uses no force variant, so the branch is rebuilt as `row/LTX25-TOKEN-APPEND-CLEAN` with the spec in one commit and the whole implementation, plus the `docs/` edits that pay for it, in the next. The content is unchanged: this branch's tree hash equals a1b1d8d's. Nothing here is a second attempt at the row. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…rim it back (#930) The LTX-2.5 phase loop was fixed at one `Ltx2VideoTokenCount(vshape, 1)`, and that single limitation blocked three conditioning arms. Two rows walked into it hours apart without talking to each other — #930 from the IC-LoRA port and #920 from the generated-keyframe-slots port — which is why it gets a seam of its own rather than a fix inside whichever row noticed it last. The appending conditioning ITEMS were already ported and gated. What was missing were the two halves that make an append survive the loop: `Ltx2ExtendKeyframesMask` mirrors `extend_keyframes_mask` (conditioning/mask_utils.py:74-105). Upstream's docstring makes the call an obligation of appending itself — "Every conditioning item that appends tokens must call this, otherwise the per-token marker goes out of sync with the token sequence" — so it lives inside `AppendTokens` rather than at the three call sites. Both None branches are mirrored, including the zero-fill that only `marked=true` reaches, so #920's arm finds it already correct. `Ltx2ClearConditioning` mirrors `clear_conditioning` (tools.py:88-117). Two things there are not a truncation: the denoise mask comes back ALL ONES rather than sliced (`tools.py:104`), because the returned state describes a finished latent, and `keyframes_mask` is dropped entirely (`tools.py:113`). `Ltx2LatentState` grows a `keyframes_mask` field so the marker can be extended with the sequence it describes. It does NOT grow an attention-mask field, and that is a finding rather than a shortcut: both appending video items pass a literal `attention_mask=None` (keyframe_cond.py:68-76, reference_video_cond.py:88-96), `update_attention_mask` returns None for that case, and the only route to a non-None mask is `ConditioningItemAttentionStrengthWrapper`, applied solely at iclora_utils.py:169 on the IC-LoRA path. A field here would be one no ported item could populate. The refusals that cite the absent field were correct to mention it and wrong to offer it as the blocker. The sigma schedule now reads a `target_tokens` local rather than `video.tokens`. That call sits after the conditioning block, so before this change it would have re-shifted the whole trajectory the moment anything appended. Upstream fixes the count at the target twice over: the argument is `math.prod(latent.shape[2:])` of the UNPATCHIFIED target (schedulers.py:32), and every pipeline computes its sigmas before a state exists (ti2vid_one_stage.py:207, distilled.py:200-201). The LAST-frame keyframe arm is lifted as the demonstration, mirroring `combined_image_conditionings` (helpers.py:272-308): one preprocess-and-encode shared by both arms, and a branch on `frame_idx` that sends 0 to `VideoConditionByLatentIndex` and anything else to `VideoConditionByKeyframeIndex`. Both may be supplied at once. Reference video, reference audio and generated keyframe slots stay refused, and the reference-video message is byte-identical. At this base PR #938 is open and unmerged, so `--lora` still does not read the IC-LoRA scale factors and that refusal's stated cause is still true. #930's body describes the message as already rewritten onto token-append; it is not. Rewording it would have shipped a refusal naming the first of two causes, which is the defect a sibling row nearly shipped. The witness is on RENDERED BYTES with a no-op control, because `Ltx2ConditioningTrace` is filled before denoise and cannot see what the loop does. A keyframe render differs from a no-keyframe render in 59773 of 91169 artifact bytes, two different keyframes differ from each other in 334, and the same request twice is byte-identical. The control is what separates "the append reached the maths" from "the instrument is blind" — a sibling row's first attempt found every arm identical INCLUDING its control, and without that control a blind instrument reads as a real-but-subtle effect. TWO ADVERTISED GUARANTEES HAD NO MUTATION THAT COULD MOVE THEM, and the fresh review that found them is folded in here rather than left as a later commit. M6 — `Ltx2ClearConditioning` slices the mask instead of restoring all ones — was GREEN. `Ltx2CreateVideoLatentState` already fills every target token's mask with 1.0, and the append writes `1 - strength` only at the TAIL, which the trim drops under either implementation. Over the range the loop walked, slice and restore produce identical bytes, so the case that carries the header's "TWO THINGS HERE ARE NOT A TRUNCATION" claim was asserting a value nothing could change. The case now conditions a token INSIDE the target first, through `Ltx2ConditionVideoByLatentIndex` at `latent_idx = 0`, which writes `1 - strength` at `start .. start + count` (latent_cond.py:41). A slice leaves 0.4 at token 0 and the loop REDs. It also REQUIREs that 0.4 BEFORE the trim, so a later change that stops arming the instrument fails loudly instead of quietly restoring the blind state. M10 — the last-frame arm's `frame_idx` becomes 0 instead of `frames - 1` — was GREEN. Both renders still differed from the no-op control and from each other, and the token count was identical, because a keyframe pinned to the FIRST frame appends exactly as many tokens as one pinned to the last. The pixel witness can see THAT an append happened and WHAT was appended; nothing could see WHERE, which is the entire content of the arm this row exists to lift. The engine now asserts the first appended token's temporal position, recomputed from `frames` and `fps` rather than read back from the `frame_idx` argument, so the check and the thing it checks are independent expressions (keyframe_cond.py:52-58). The citation sweep behind the anchors was redone from scratch rather than trusted, deriving each needle from the CLAIM instead of reading it out of the cited span. It covers 117 citations in this row's added lines — 107 explicit and 10 bare `:NN` continuations. `ltx2_recipes.py:125-158` cites a file that exists neither upstream nor here; the distilled recipe's frozen sigmas are distilled.py:200-201 over utils/constants.py:17-23. A bare `(:100-101)` in the extend-mask case resolves against `tools.py`, the nearest file named above it, whose :100-101 is a real and unrelated statement; it means mask_utils.py. The two bare continuations in `Ltx2ClearConditioning`'s declaration are now spelled with their file for the same reason: the nearest name above them is `blocks.py`, and blocks.py:104 and :113 are both real imports. `Ltx2ExtendKeyframesMask(marked=true)` lands with a unit driver and no product caller. It is named in the pull request body and in the spec's `## Owed`, owned by row `LTX25-GENERATED-KEYFRAMES` and tracked by #920. WHY THIS CODE LANDS TWICE IN `git log`. It was first committed on `row/LTX25-TOKEN-APPEND` (PR #948, head a1b1d8d) and reviewed to PASS there. That branch cannot land: `scripts/check-doc-checkpoint.py` walks a range PER COMMIT (#573), and the review-repair commit b5618b3 edited `include/vllm/` headers — `user_usage` — while the `docs/USAGE.md` edit that pays for them sat in a sibling commit. The final tree satisfied the obligation and only the ordering failed. A commit message cannot be corrected in place and this repository uses no force variant, so the branch is rebuilt here with the whole implementation and both `docs/` edits in one commit. This is not a second attempt at the row: the tree at this commit is byte-identical to a1b1d8d's, proven by tree hash. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
localai-bot
pushed a commit
that referenced
this pull request
Aug 16, 2026
…the first, and it came from main (#968, #672) With #965's `C4456 'loaded' shadow` removed, `windows-msvc-cpu` and `windows-msvc-vulkan` failed again on this row's pull request — and on a different cause: include\vector(1461,29): error C2220: the following warning is treated as an error include\vector(1461,29): warning C4244: '=': conversion from 'const double' to 'float', possible loss of data raised from `src/vllm/multimodal/ltx2_video.cpp:203,214`, the two narrowing `positions.assign` calls that `c7cb59fbb` (#964, LTX25-TOKEN-APPEND) landed on `main` while this row was in flight. `StreamState::positions` and `Ltx2LatentState::positions` differ in element type; GCC and Clang narrow silently, MSVC diagnoses and the build treats it as an error. NOT FIXED HERE, deliberately. #964's own comment at `ltx2_video.cpp:129-132` reasons that "double -> float -> double reproduces the bits", so the narrowing is intentional and a silencing `static_cast` would be a claim about that reasoning rather than a formatting repair. It belongs to the lane that owns the round trip. Filed as #968 with the evidence rather than papered over. THE MATCHED ARM SPLITS EXACTLY ON THE MERGE BASE, which is what makes it inherited rather than mine. Grepping each `windows-msvc-cpu` job log for the warning: #966 and #951, both on `c7cb59fbb`, hit it twice each; #967, #956, #950, #939 and #938, all based before it, do not hit it at all. This row's diff touches zero LTX2 files. THE FINDING WORTH CARRYING is not either warning. It is that TWO INDEPENDENT CAUSES WERE STACKED BEHIND ONE HABITUALLY-RED JOB NAME, and the first hid the second — and that neither was #645, the `M_PI` regression both jobs are usually attributed to. A known-red list tells you a job is often red. It never tells you that today's red is the same one. Only reading the log does. Issue: #968 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
localai-bot
pushed a commit
that referenced
this pull request
Aug 16, 2026
Main advanced to c7cb59f, whose #964 (LTX25-TOKEN-APPEND) edits the same three files this row does: `src/vllm/multimodal/ltx2_video.cpp`, `docs/USAGE.md` and `docs/FEATURES.md`. Git reported no textual conflict, which is not the same as a correct merge for a keyed record, so both documents were re-derived by key rather than accepted from the three-way result. `docs/USAGE.md` differs from `origin/main` by exactly one hunk, and the four lines it removes are byte-identical to the four this branch removed from the merge base. #964's two edits, the keyframe paragraph near line 501 and the `--offload-config` row of the flags table, survive untouched. `docs/FEATURES.md` differs from `origin/main` by one added row and no removed line, so no key #964 wrote is disturbed. That row's widest cell is 212 characters against the 220 the checker refuses above; #964 pushed the LTX-2.5 DiT row's last cell to exactly 220, which passes because the comparison is strictly greater. The READER ANCHORS list in `ltx2_video.cpp` needed no edit and is not assumed to: re-derived from the merged tree by the same walk `test_ltx2_video` performs, the nine readers sit at 756 811 907 923 925 1003 1028 1133 1174, which is the list the merge took from main. This branch's additions to that file all fall after line 1174, so they move no reader. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
localai-bot
added a commit
that referenced
this pull request
Aug 16, 2026
…publish the LTX-2.5 envelope (#939) Nothing in this tree established what resolutions LTX-2.5 supports. `ltx2-gen` exposes `--width`, `--height` and `--frames`, and exposing a flag is not supporting a value. Reading the pinned oracle answers the three axes differently, and that difference is the finding. Closes #919. Row `LTX25-RESOLUTION-ENVELOPE`, spec [`.agents/specs/ltx25-resolution-envelope.md`](.agents/specs/ltx25-resolution-envelope.md). Upstream pin `Lightricks/LTX-2` @ `fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca`, verified at the local checkout before any anchor was read. ## The defect `vllm_video_generate` integer-divided `width` and `height` into the latent grid and checked only the lower bound. Measured on the reduced fixture at the base SHA, before the guard existed: | Request | Recipe | What happened | |---|---|---| | width 80 | distilled two-stage | rendered **64x64**, exit success | | width 100 | one-stage | rendered **96x64**, exit success | | width 96 | distilled two-stage | threw `the upsampled latent is 4x2x2x2 but phase 'refine' needs 4x2x2x3` | The third is not a silent floor, but it is not a usable error either: stage 1 floors 48 to one latent cell while stage 2 needs three, so the upsampler's shape check fires with a true statement about latents and no mention of the width the caller passed. One guard at the entry point closes both faces, which is why the test carries all three sizes. Our own `docs/USAGE.md` already documented the divide-by-64 rule as though something enforced it. ## Upstream raises, so we refuse `assert_resolution` (`ltx-pipelines utils/helpers.py:540-551`) raises `ValueError`, from the top of a pipeline's `__call__` and before any work is paid for. **Nine invocations**, including `ti2vid_two_stages.py:184` and `ti2vid_two_stages_hq.py:199` (`is_two_stage=True`) against `ti2vid_one_stage.py:156` (`False`). Nine, and not the ten an earlier revision of this description claimed, nor the twenty-one lines a grep for the name returns: those are 9 invocations + 1 definition + 10 imports + 1 `__all__` string. Nor is it every pipeline. 13 pipeline `__call__`s take a height and a width, and the four that do not call the guard are `distilled_mgpu.py:143`, `ti2vid_two_stages_mgpu.py:163`, `ti2vid_two_stages_hq_mgpu.py:164` and `hdr_ic_lora.py:352`. The count was established by walking every `def __call__` and reading its signature, because the grep that produced "ten" counts imports. **The divisor is derived, not restated.** Upstream spells 64 and 32 as literals chosen by a bool, but that pair is the VAE spatial factor (32, `ltx_core/types.py:31-33`) times the worst spatial downscale any phase applies: a two-stage pipeline runs stage 1 at `width // 2` (`ti2vid_two_stages.py:226-228`), so a request must survive being halved and still divide the grid. `Ltx2AssertResolution` takes the divisor as a parameter and the call site computes `factors.height * recipe.max_spatial_downscale()`, which reproduces upstream's two numbers on the two shipped arms and stays correct for a recipe whose phases downscale further. Two subcases drive the same width 96 through both recipes and require opposite answers, so a hardcoded 64 fails the suite. One divisor covers both axes, as upstream has it. That is the mirror rather than a simplification, and the call site asserts `factors.height == factors.width` so a VAE that ever broke the assumption fails by name instead of having its width measured against the height factor. ## Frames are the opposite answer `--frames` is deliberately left rounding, and the asymmetry is upstream's. `resolve_num_frames` (`utils/blocks.py:908-928`) returns an explicit count verbatim (`:920-921`), and `VideoLatentShape.from_pixel_shape` (`ltx_core/types.py:113`) then floors it exactly as this engine does. Adding a refusal there would diverge from the reference rather than mirror it, so the repair is to the document that promised enforcement. **The premise this rested on was false, and the replacement is checkable.** An earlier revision said `snap_frames_to_grid` (`utils/helpers.py:554-562`) "is reached only from the auto-duration path". It has three callers, not one: `utils/helpers.py:581` inside `seconds_to_clamped_num_frames`, which is the auto-duration path, and `dubit.py:215` and `:396`, the second three lines after `DubitPipeline.__call__`'s own `assert_resolution`. What actually holds is sharper and can be checked against a signature rather than a call graph: `DubitPipeline.__call__` takes **no `num_frames` parameter at all** (`dubit.py:194-210`) and snaps a count it read from a reference video's container metadata. It is the only pipeline `__call__` that snaps, and the only one with no frame-count parameter. Every `__call__` that does take one leaves it unsnapped. ## The published envelope `docs/USAGE.md` gains "The supported resolution envelope", which separates what is legal from what fits: legal sizes are any multiple of 64 or 32; upstream's own defaults are 1024x1536 and, for the HQ preset, 1088x1920 at 121 frames; and the measured ceiling on one GB10 is 320x192 at 25 frames, with 448x256 losing about 59 GB inside the decode. That gap is a decode problem and not a cap. There is no maximum-size check anywhere in this path, and the 60 GB is explicitly **not attributed** — the decode's own heap peak at that size is 361.72 MiB. ## Tests **No upstream test is ported, because there is none to port.** `Lightricks/LTX-2` at `fd4ded7f` contains zero `test_*.py` files anywhere in the repository. The tests are written against upstream anchors instead. Only the provenance changes: each fails for the intended reason before the change, enters through `vllm_video_generate` rather than constructing the type, and names the upstream `file:line` justifying the behaviour it asserts. **Reachability.** All three production entry points reach the guard, traced by hand and re-derived at the head below rather than carried forward: `include/vllm.h:969 vllm_video_generate` → `src/capi/vllm_c.cpp:1646 engine->engine->Generate(gen)`; `examples/ltx2_gen/main.cpp:320 vllm_video_generate(...)` through the same ABI (it includes `vllm.h` and nothing internal); and the OpenAI `/v1/videos` route through `src/vllm/entrypoints/openai/server_main.cpp:1292 video_engine->Generate(...)`. All land in `Ltx2VideoEngine::Generate` (`ltx2_video.cpp:1210`) and hit the guard at `:1547`. **Three mutations were re-run at `59d4f59ca`, the head this description belongs to**, because two of them had been claimed by construction and never executed, and the third had not been re-run since the last merge. Each carries three facts — the diff after applying, whether it BUILT with the compile-error count, and the exit code — because a mutation that fails to build establishes nothing: | Mutation | Diff | Built | Exit | Result | |---|---|---|---|---| | Swap the two axis names in the refusal | 1 file, +2/-2 | YES, 0 errors | 1 | RED — 10 of 33 assertions fail | | Suggestion back to the bare floor (the `0x64` a sub-divisor caller used to get) | 1 file, +2/-6 | YES, 0 errors | 1 | RED — 5 fail, headed by `CHECK(msg.find("0x64") == npos)` | | Delete the production call site | 1 file, +1/-1 | YES, 0 errors | 1 | RED — the case aborts on its first `FAIL` | The swap is the mutation that mattered, and it was **green** before this repair. The message emits the literal `" (width x height) "` label in every refusal it ever produces, so `msg.find("width")` was satisfied by that constant no matter which axis the guard named: a height-80 request could have reported "the width is not" with nothing to see it. The needles are now whole phrases (`the width is not` / `the height is not` / `the width and height are not`), every subcase asserts the other axis **absent**, and an 80x80 subcase covers a branch no test executed. Each mutation was restored by writing the original bytes back and comparing `sha256` (identical in all three cases, `git status` clean), and the tree was rebuilt before any measurement was taken afterwards. **The case name contains a comma, and `-tc` treats a comma as a filter separator.** `-tc="*is REFUSED, per recipe*"` does not select this case: it selects three unrelated ones, runs 8 assertions instead of 33, and prints `SUCCESS`. Every run above therefore used `-tc="*does not divide the latent grid*"` and asserted a non-zero case count; the baseline is 1 case / 33 assertions / exit 0. ## Gate Clean-room: `build/` did not exist and was configured from scratch for this run. Run at this branch's head, `59d4f59ca`, on a **loaded** box — the load is recorded beside every number rather than left to be assumed: ``` CONFIGURE_EXIT=0 loadavg 15.89 BUILD_EXIT=0 ': error:' = 0 'No space left' = 0 'BFD assertion' = 0 ctest -N Total Tests: 485 CTEST_EXIT=0 100% tests passed, 0 tests failed out of 485 (2 skipped) Total Test time (real) = 270.86 sec loadavg 146.38 at start, 50.36 at end ``` The two skipped are `test_modelopt_mixed_precision_checkpoint` and `test_voxtral_e2e`, both skipped by their own guards. Nothing on this run needed a serial re-run: the load-dependent cases (`test_openai_conformance`, `test_cpu_threadpool`, `test_engine_core_proc`, `test_async_llm`, `test_cpu_x86_llamacpp_floor`) all passed under `-j4` at loadavg 146. **The three zero counts are measurements, not a broken instrument.** Each `grep` pattern was positive-controlled against a synthetic log carrying all three strings: `No space left` → 1, `BFD assertion` → 1, `: error:` → 0 before the error line was appended and 1 after. The binary was also checked for probe residue (`strings … | grep -c PROBE` = 1, and that one hit is `VT_H3_VAE_PROBE`, a committed env-var name in `minimax_h3_pipeline.cpp:581`, not residue), and its mtime post-dates the rebuild that followed the last mutation restore. `scripts/agent-integration.py --base origin/main` reports **all gates green** locally, including `check-agent-record`, `check-doc-checkpoint`, `check-now-current`, `check-public-doc-tables`, the trailer suites and the commit-style suites. `check-doc-checkpoint.py` walks **per commit** (#573), so it was run that way on all ten commits in `origin/main..HEAD` before the push: all ten exit 0. Positive control: `--commit b5618b3` exits 1, so the checker was armed. That per-commit walk found one real defect and it is repaired here. The recovered repair commit edits `src/vllm/model_executor/models/ltx2_pipeline.cpp` — a `feature_surface` path — while the `docs/FEATURES.md` row it owes had landed in an earlier commit on the branch. The range check was green and the per-commit check was red, which is exactly the failure mode #573 describes. The `docs/FEATURES.md` edit now rides in that commit, and it is a real one: the State cell said "refused by name", which overstated what the message gave back to a sub-divisor caller. ### CI `windows-msvc-cpu` and `windows-msvc-vulkan` are known-red on every pull request and have no `main` baseline (#584). **`sanitize-cpu` and `agent-record` are NOT in that category any more** — #873 and #904 are fixed on `main` — so a red in either on this head is new information and is not waived here in advance. **No GPU was used.** ## Owed - #921 — the res_2s **denoising loop** (`samplers.py:208-447`) is unported, so `TI2VidTwoStagesHQPipeline` cannot be served. What exists is one substep's SDE arithmetic (`Ltx2Res2sStep`/`Ltx2Res2sSdeCoeff`, gated). Absent are the `phi`/`get_res2s_coefficients` exponential integrator (`res2s.py:1-60`), the second transformer evaluation per step at `sub_sigma = sqrt(sigma * sigma_next)`, the bong anchor refinement, and any `Ltx2StepperKind` enumerator to select it. No HQ recipe row is added here, so nothing can select it and nothing lands dead. Serving the HQ preset on the Euler loop would render a plausible clip that is quietly not HQ. - `TI2VidTwoStagesPipeline` as a distinct recipe row — covered by #644. - Attributing the 60 GB decode loss, and the single-threaded decode throughput. Both need the GPU. - **The lcm form of the divisor.** `max_spatial_downscale()` takes the maximum where the quantity a request must survive is the least common multiple of the phase downscales. The two agree on every shipped recipe, whose downscales are 1 and 2, and part on a recipe with phases at 2 and 3: the max gives 96, a 96-wide request passes, and the downscale-2 phase then floors 48 onto one latent cell. This is **narrowed and recorded rather than implemented**, because no production entry point can reach the difference today and no test entering through one could gate it. The limit is stated in the header comment on `Ltx2AssertResolution` so the recipe row that adds a non-power-of-two downscale finds it, and under `## Owed` in the spec. ## Records `docs/FEATURES.md` gains **one appended row and nothing else**: `git diff origin/main -- docs/FEATURES.md` is `1 0`, one insertion and zero deletions, so no key `origin/main` wrote is disturbed. The spec had declared the file out of scope on the grounds that a refusal changes no feature surface; `check-doc-checkpoint` disagreed and was right, and the spec records the correction. Git auto-merged this keyed record without a textual conflict, which is not the same as a correct merge, so it was re-derived by key rather than trusted. That row's widest cell is 212 characters against the 220 `check-public-doc-tables.py` refuses **above** — #964 pushed the LTX-2.5 DiT row's last cell to exactly 220, which passes because the comparison is strictly greater, and this row was written to fit under the cap on its own rather than by trimming somebody else's wording. `docs/USAGE.md` differs from `origin/main` by **exactly one hunk**, and the four lines that hunk removes are byte-identical to the four this branch removed from the merge base. #964's two edits to the same file — the keyframe paragraph and the `--offload-config` row of the flags table — are untouched. The `READER ANCHORS` list in `ltx2_video.cpp` was re-derived from the merged tree by the same walk `test_ltx2_video` performs, not assumed: the nine readers sit at `756 811 907 923 925 1003 1028 1133 1174`, which is what the merge took from `origin/main`. This branch's additions to that file all fall after line 1174, so they move no reader, and the gate that would have caught it otherwise passes. `.agents/issue-index.md`: two rows appended, zero removed. GitHub first reported this pull request `CONFLICTING` while `git merge-tree` reported the same pair clean, which is **#883**: the union driver `.gitattributes` declares for that path is a local driver and the forge does not run it. The index was the only file both sides touched, and both sides only appended. Merging `origin/main` here applied the driver and left the forge nothing to resolve. After the merge of `c7cb59fbb` the index was re-verified two ways: `git diff origin/main -- .agents/issue-index.md` shows exactly the two rows this branch appends and no removal, and a scan of every row key finds no duplicate, which is the shape a union merge of two relocations produces. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
localai-bot
pushed a commit
that referenced
this pull request
Aug 16, 2026
…our conflicts by hand (#920) The branch was fifteen commits behind. Four files conflicted, and two of them are keyed records, so none of the four took an automatic three-way merge. The merged commit is named as a SHA rather than as `origin/main`, because `origin/main` moved between the first attempt at this merge and the second one. The first resolution was built against `c7cb59fbb`, and `e5351776c` (#939) landed while it was being written, rewriting 54 lines of the very `docs/USAGE.md` section this row edits. Merging the moving ref would have carried a resolution built against a tree that no longer existed. `include/vllm/multimodal/ltx2_video.h` and `src/vllm/multimodal/ltx2_video.cpp` conflicted with row LTX25-A2V-AUDIO-INPUT (#922), which added three per-generation extras beside this row's one. Both sides are additive and neither edits the other's text: the header takes main's `audio_path`, `audio_start_time` and `audio_max_duration` block followed by this row's `num_generated_keyframes` block, and diffing the resolved header against `e5351776c` gives 33 added lines and 0 removed. The `.cpp` extras check takes main's `known` predicate with this row's key added as a fifth disjunct and a fifth name in the message, rather than this branch's two-key `!=` chain, which main had already replaced. `docs/FEATURES.md` and `docs/USAGE.md` are keyed records, so this commit takes main's version of both BYTE-FOR-BYTE and re-applies nothing: `git diff e535177 -- docs/` is empty. The scoped edits belong to the repair commit that follows, because both were written against a tree where a supplied last-frame keyframe was refused, and #964 landed the seam that serves it. Re-applying them here would carry a claim this merge already knows to be false, and the FEATURES cell has no room for it either: the LTX-2.5 row's fourth column is 220 characters on main, which is `MAX_CELL_CHARS` exactly. The READER ANCHORS comment in `ltx2_video.cpp` was re-derived at this tree and is unchanged at `756 811 907 923 925 1003 1028 1133 1174`. Main's values carry, because every line this row adds sits below 1174, which is what the row's `## 7` predicted and what the gate now confirms rather than assumes. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
localai-bot
added a commit
that referenced
this pull request
Aug 16, 2026
… heard, and the five keys upstream refuses that we dropped (#672, #953, #965) (#966) feat(MODEL-MUSIC-MUSIC3): a music-only server, an example that can be heard, and the five keys upstream refuses that we dropped (#672, #953) Developer directive: parity on what upstream supports — "we want to be a good reference" — usage docs for MiniMax-Music3, and in those docs the weights. Then, mid-flight: "we should allow to load only the music model" and "we need to have an e2e test working". FOLLOWING_AGENTS_PROTOCOL ## The upstream surface, enumerated rather than summarized SGLang-Omni `748a0b43` `models/minimax_music3/` and diffusers `c6da9936` `modular_pipelines/minimax_music3/`, read field by field and recorded with `file:line` in spec §10.1 so the next reader re-derives nothing. **Closed here:** the music-only server, the missing example, and five refusals. **Owed and named:** the non-`wav` response formats, request batching and `/v1/audio/speech/batch`, the 32 kHz delivery resample. **Permanently refused rather than owed:** streaming — neither upstream arm has it (`supports_streaming_vocoder=False`). **One place we are ahead of both arms:** `guidance_scale` is a real per-request control here, where diffusers freezes it at 1.7 into the guider component (`denoise.py:180`) and SGLang exposes it only as a serve-time knob. ## `--model` is optional when `--speech-model` is given Serving a 28.5 GB music model also forced loading an unrelated text model, and on this box the smallest text checkpoint is 35B — so the recipe this project documented was effectively unrunnable. Upstream's own is `sgl-omni serve --model MiniMaxAI/MiniMax-Music3`, no text tower anywhere. vllm-server --speech-model /path/to/minimax-music3 Third instance of a shape already in `server_main.cpp`: a pooling checkpoint serves `/v1/embeddings` alone, a Parakeet checkpoint serves `/v1/audio/transcriptions` alone. It mirrors vLLM's task-conditional registration (`api_server.py:255-265`). **Additive, and proved rather than argued.** The only case whose verdict changes is BOTH flags absent, which was an error and remains one with a message naming both options. The route table is gated in both directions over a real socket, because a handler-dispatch test cannot see route registration at all. ## The example the music family did not have `examples/minimax_music3_gen` — a thin client of `include/vllm.h` and nothing else, like `parakeet-transcribe` and `vllm-cli`. Hearing this model previously needed a running server plus a `curl`, or a C ABI caller nobody had written. ## Five keys upstream refuses by name were SILENT here (#953) `temperature`, `top_p`, `top_k`, `repetition_penalty` — refused upstream at `request_builders.py:14-19,109-114`, because this model's autoregressive stage has ONE sampler, a fixed top-50 draw (`encoders.py:48,94-103`). And `max_new_tokens`, upstream's LENGTH spelling in 25 Hz frames rather than seconds (`request_builders.py:56-68`), so a 250-frame request silently became the family's 60 s default. That is the #925 class exactly, in the same file that already carries #925's refusal one paragraph above. Fixed in flow. ## The e2e gate no longer reports a skip wearing a pass It read 5 cases / 5 passed with **`assertions: 0`** whenever the checkpoint was absent — the same shape that fooled this project on `test_qwen3_paged_engine`. Split into a checkpoint-free half that runs unconditionally in CI (request contract, both ceilings, the speech-only route table over a real socket with a stub synthesizer) and the env-gated half, whose HTTP case now drives the real engine over a real socket against the music-only server shape. A coverage-report case prints which arms ran, every run. The full arm was run: `POST /v1/audio/speech -> 200 audio/wav, 12332 bytes in 518 s wall`, 2 AR frames -> 6 latent frames -> 3072 samples/channel, 6144 int16 samples all non-zero, 0 clipped, 2818 of 3072 positions differing between L and R, and `/v1/completions` + `/v1/chat/completions` both 404 from the route table. `checkpoint_arms_run=5`. | arm | cases | assertions | |---|---|---| | `test_minimax_music3_e2e_real`, no env vars | 9 | 37 (was 5 / **0**) | | `test_minimax_music3_e2e_real`, checkpoint only | 9 | 86 | | `test_minimax_music3_e2e_real`, checkpoint + `VLLM_CPP_MUSIC3_DIT=1` | 9 | **582** | | `test_speech_api` | 6 | 67 | | `test_openai_api_server` | 62 | 733 | | `test_openai_conformance` | 23 | 252 | | `test_minimax_h3` (unchanged) | 79 | 57395 | | server flag ctest cases | 7 passed | | ## The weights are documented (porting-a-model.md §2.1) `docs/USAGE.md` gains component-by-component tables: the diffusers arm at `MiniMaxAI/MiniMax-Music3`@`fbdf52fbaaca799592917417eb05f1899f1255ec`, **28.5 GB resident** (28 517 617 303 B, measured) out of a 57.4 GB repository and why they differ; the native `.pth` arm we refuse and SGLang-Omni serves; the one implemented GGUF Q4_K artifact with its sha256; and the fourteen third-party quantized repositories in five formats, each marked refused. The revision is verified rather than copied — `condition_encoder/diffusion_pytorch_model.safetensors` on disk hashes to that revision's own LFS record. ## A sample a human can hear 2.0 s of 44100 Hz stereo from this engine in 3286 s of wall clock: RMS 0.03169, peak 0.97437 with 0 clipped samples, 84 073 of 88 064 positions differing between left and right. **Its samples are compared to nothing** — §5 withdrew the token gate — so it shows the pipeline runs, not that the music is right. It is not committed: `check-pr-size.py` classifies every path and none takes a `.wav` outside `tests/`, where a file compared to nothing would sit beside the goldens and imply it was one. ## The four asks, answered directly **1. Music-only server.** `vllm-server --speech-model <dir>` with NO `--model` starts and serves, observed live rather than inferred: server: speech/music-only model (family=minimax-music3, 44100 Hz, text-only synthesis, family DETECTED); serving /v1/audio/speech server: listening on http://0.0.0.0:18923 (model 'minimax-music3') `--model` alone and `--model` + `--speech-model` are **byte-identical in behaviour**. The whole change is one new early branch, `if (args.model_dir.empty())`, which loads the speech engine and `return 0`s before reaching a single line of the existing path; nothing downstream of it was touched. The only case whose verdict changes is BOTH flags absent, which was an error and remains one. Server suites: **7 of 7** ctest cases (4 pre-existing + 3 new — neither flag is still an error and now names both options; `--speech-model` alone reaches the speech LOAD; `--speech-family` alone still demands a checkpoint), `test_openai_api_server` **62 cases / 733 assertions** (+1 case / +24 assertions, the speech-only route table over a real socket), `test_openai_conformance` **23 / 252** unchanged. **2. e2e, three arms.** What a bare CI run executes unconditionally: the request contract on the exact body the real case posts, the near-miss and sampling refusals, the duration arithmetic including both ceilings, and the speech-only route table over a real socket with a stub synthesizer. What stays env-gated: everything needing the 28.5 GB checkpoint, plus the two 2.4B-DiT arms behind `VLLM_CPP_MUSIC3_DIT`. | arm | cases | assertions | checkpoint arms run | |---|---|---|---| | no env vars (what CI runs) | 9 | **37** | 0 — was 5 cases / **0 assertions** | | `VLLM_CPP_MUSIC3_CHECKPOINT` | 9 | **86** | 3 | | + `VLLM_CPP_MUSIC3_DIT=1` | 9 | **582** | 5 | **3. The five keys.** All five were **accepted and silently dropped**; all five are **now refused by name**. | key | upstream anchor | why it cannot be honoured | |---|---|---| | `temperature` | `request_builders.py:14-19,109-114` | the AR stage's only sampler is a fixed top-50 draw, `encoders.py:48,94-103` | | `top_p` | same | no nucleus branch exists | | `top_k` | same | `_AR_SAMPLING_TOP_K` is a module constant of 50 | | `repetition_penalty` | same | no penalty is applied anywhere in the loop | | `max_new_tokens` | `request_builders.py:56-68`, `constants.py:4-5` | upstream's LENGTH, in 25 Hz frames not seconds; the refusal names `audio_duration` and the /25 conversion | **4. The weights table** (`docs/USAGE.md`, "MiniMax-Music3: the exact weights"). It carries: repo **and** revision — `MiniMaxAI/MiniMax-Music3`@`fbdf52fbaaca799592917417eb05f1899f1255ec`, verified rather than copied, since `condition_encoder/diffusion_pytorch_model.safetensors` on disk hashes to `83179c5e…a202c2a4d`, that revision's own LFS record; the Q4_K artifact's sha256 `4c5d41b2…c70cbdd0` at revision `c36aaeed…` with its exact byte count; **28.5 GB resident (28 517 617 303 B, measured) versus 57.4 GB repository**, with the reason they differ; the refused native `.pth` arm (`qwen_7B/`, `flowmatching_vae.pth`, `dav.pth`) and that SGLang-Omni serves it; and all fourteen community quant repositories across five formats, each marked refused and each marked **third-party** rather than first-party. This is the first application of `.agents/porting-a-model.md` §2.1 (landing as #951). ## Two reds stacked behind one habitually-red job name `windows-msvc-cpu`/`windows-msvc-vulkan` are usually attributed to #645 (`M_PI` in three LTX2 sources). **Neither of the two causes here was #645**, and the first hid the second. **#965, fixed in flow.** `C4456: declaration of 'loaded' hides previous local declaration` at `server_main.cpp:1315` — W6's own speech-attach block declaring `loaded` inside the text engine's `loaded`. The only warning in the job, on `main` since W6 landed. Matched arm: #956, #950 and #939, none touching the speech surface, fail identically. Renamed; nothing suppressed. **#968, filed and NOT fixed here.** With the shadow gone the same jobs failed again on `C4244: conversion from 'const double' to 'float'`, raised inside MSVC's `<vector>` from `ltx2_video.cpp:203,214` — two narrowing `positions.assign` calls that `c7cb59fbb` (#964) landed on `main` while this row was in flight. **This branch touches zero LTX2 files.** The matched arm splits exactly on the merge base: #966 and #951 (on `c7cb59fbb`) hit it, #967/#956/ #950/#939/#938 (before it) do not. It is deliberately left to the LTX-2.5 lane — #964's own comment reasons that "double -> float -> double reproduces the bits", so a silencing cast is a claim about that reasoning rather than a formatting fix. **The finding, which outlives both:** a known-red list tells you a job is often red. It never tells you that today's red is the same one. Only reading the log does — and here it took two readings, because removing the first cause is what made the second visible. <!-- kept for the record --> ### The first of the two, in detail (#965) `windows-msvc-cpu`/`windows-msvc-vulkan` failed here, and they are **not** #645 (`M_PI` in three LTX2 sources). They were W6's own `C4456: declaration of 'loaded' hides previous local declaration` at `server_main.cpp:1315` — the only warning in the job, on `main` since W6 landed. The matched-arm check is what separated it from my diff: #956, #950 and #939, all touching no speech surface, fail identically. `main` has no baseline because `windows-msvc-*` are PR-only (#584), so it presents to each author in turn as their own red. Filed and fixed in flow by renaming the inner declaration; nothing suppressed. ## Mutations Four run, four fire: sampling refusal neutered (5 assertions red), `max_new_tokens` refusal neutered (2 red), `--model` made mandatory again (2 ctest cases red), generate routes registered unconditionally (3 cases / 6 assertions red in the api-server suite, 1 / 2 in the e2e suite). Sources restored and verified sha256-identical. Supersedes #954 (untrailered merge commits) and #963 (a `server_main.cpp` commit that owed `docs/USAGE.md` under the per-commit documentation checkpoint). Same tree, linear history, every commit green on `check-doc-checkpoint`, `check-commit-trailers` and `check-commit-style` locally before pushing. Every source file is byte-identical to the one built and gated. Issue: #672 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
localai-bot
pushed a commit
that referenced
this pull request
Aug 16, 2026
…ady landed, and its test could not tell (#975) A fresh review returned FAIL on three findings, and the largest of them is that the refusal this row rewrote was FALSE before it merged. ## The refusal named a cause the tree had closed Row `LTX25-IC-LORA` moved the reference-video refusal off the IC-LoRA metadata, which it had genuinely closed, and onto the token-APPEND machinery. That was accurate on 2026-08-15. Row `LTX25-TOKEN-APPEND` landed the seam in `c7cb59fbb` on 2026-08-16, before this branch merged, so both reasons the message had ever given were false at once. `#964` left the wording byte-identical only because this pull request was open and its cause was then still true. The determination was made again from the merged tree rather than inherited. The attention-strength wrapper is not in the way either: on the DEFAULT arm upstream sets `attn_mask = None` at `conditioning_attention_strength >= 1.0` with no latent mask (`iclora_utils.py:159-160`) and applies `ConditioningItemAttentionStrengthWrapper` only `if attn_mask is not None` (`:168-169`), so #932 is not the blocker for the default case. TWO causes remain, and the refusal now names both. 1. The reference CLIP has no pixel path. Upstream reads it at `height // scale` by `width // scale` (`iclora_utils.py:116-117`), refuses a target the factor does not divide (`:112-115`), keeps frame 0 then every Nth frame (`temporal_subsample`, `:87-89`, called at `:144`) and encodes the whole clip (`:145-148`). This engine's only pixel-to-latent route encodes exactly one frame at the phase's own resolution, and nothing reads `ref_video_dir`. 2. The reference item is a STAGE-1 item and stage 2 must run UNFUSED. `ic_lora.py:108` gives stage 1 `loras=tuple(loras)` and the reference conditioning (`:269-278`, `:377-402`); `:119` gives stage 2 `loras=()` and `:314-321` gives it `combined_image_conditionings` with no reference item. This engine holds ONE DiT, fused at load, that every phase runs. Serving the arm is therefore its own row, filed as #975. Piece 2 changes how the engine holds its weights, not how it conditions, and a second resident DiT is a memory decision this row cannot take on its own. ## The test could not have caught any of that, and now can The case asserted five SUBSTRINGS. Two were upstream symbol names, present in the pinned checkout whatever this engine can do, and three were literals the message declared about itself. None could go red when the ENGINE changed. The reviewer proved it: replacing the local-cause sentence with a self-declared falsehood, keeping all five substrings, left `test_ltx2_video` at 44 cases / 914 assertions / SUCCESS. The case now MEASURES the engine first. It renders with and without an appending conditioning item, reads `video_tokens` — the one trace field written inside the phase loop — and requires the grown count to exceed the plain one and both renders to come back at the target frame count. Only then does it constrain the message, and the property it asserts is positional rather than lexical: every occurrence of a closed cause must sit after the `WHAT IS *NOT* THE REASON` marker, because recording a ruled-out cause is the message's own convention and must stay possible. ## The bf16 headline was only two thirds gated The aggregation dtype binds three roundings, and only two had cases: `B * strength` (`fuse_loras.py:113`) and `deltas.add_(weight)` (`:67-68`). The matmul RESULT's own `.to(dtype=dtype)` had none — widening only it left `test_ltx2_lora` at 13/13 and `test_ltx2_loader` at 31/31. The new case puts `acc = 1 + 2^-8` exactly on a bf16 tie and adds `w = 2^-9`, so the ported order stores 1.0 and an f32 accumulator stores 1.0078125, one bf16 step apart in the STORED result where the final rounding cannot absorb it. ## Records `.agents/issue-index.md` carried a SECOND `#930` row describing that blocker as open. #930 is closed, and main already carries the authoritative row, so the duplicate is dropped before it lands — the append-only rule protects rows that exist on `main`, and this one never did. The `#932` row's "blocked behind #930" is corrected, and #975 is appended. FOLLOWING_AGENTS_PROTOCOL Issue: #975 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
localai-bot
added a commit
that referenced
this pull request
Aug 16, 2026
…, and stop blaming the metadata (#923) (#938) Closes #923. Row `LTX25-IC-LORA`, spec [`.agents/specs/ltx25-ic-lora.md`](.agents/specs/ltx25-ic-lora.md). Upstream pin `Lightricks/LTX-2` @ `fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca`, verified at the local checkout with a clean tree before any anchor was read. Upstream `ICLoraPipeline` (`ltx-pipelines/ic_lora.py`) is video-to-video on the distilled model. This row builds the half of it that was actually missing — the adapter path — and leaves the other half refused, on the causes that genuinely remain. ## The reference refusal named a seam that had already landed This is the finding a fresh review returned FAIL on, and it is the most important thing in this pull request. The refusal originally blamed the IC-LoRA metadata. That was true, and this row closed it. The row then rewrote the refusal onto the **token-APPEND machinery**, which was accurate on 2026-08-15 and false on 2026-08-16: row `LTX25-TOKEN-APPEND` landed that seam in `c7cb59fbb` while this pull request was open, and the LAST-frame keyframe arm is served on it today. #964 left the wording byte-identical only because #938 was open and its cause was then still true. Both reasons this message had ever given were false at once. The determination was made **again, from the merged tree**, rather than inherited. The attention-strength wrapper is not in the way either: on the default arm upstream sets `attn_mask = None` at `conditioning_attention_strength >= 1.0` with no latent mask (`iclora_utils.py:159-160`) and applies `ConditioningItemAttentionStrengthWrapper` only `if attn_mask is not None` (`:168-169`), so #932 is not the blocker for the default case. **Two causes remain, and the refusal now names both.** 1. **The reference clip has no pixel path.** Upstream reads it at `height // scale` by `width // scale` (`iclora_utils.py:116-117`), refuses a target the factor does not divide (`:112-115`), keeps frame 0 then every Nth frame (`temporal_subsample`, `:87-89`, called at `:144`) and encodes the whole clip (`:145-148`). This engine's only pixel-to-latent route encodes exactly one frame at the phase's own resolution and refuses an encode returning more, and nothing anywhere reads `ref_video_dir`. 2. **The reference item is a STAGE-1 item and stage 2 must run UNFUSED.** `ic_lora.py:108` gives stage 1 `loras=tuple(loras)` and the reference conditioning (`:269-278`, `:377-402`); `:119` gives stage 2 `loras=()` and `:314-321` gives it `combined_image_conditionings` with no reference item. This engine holds ONE DiT, fused at load, that every phase runs. Serving the arm is filed as **#975** rather than done here. Piece 2 changes how the engine holds its weights, not how it conditions; a second resident DiT is ~21 B parameters, which is a memory decision this row cannot take on its own. ## The test could not have caught any of that, and now can The case asserted five substrings of the message. Two were **upstream** symbol names, present in the pinned checkout whatever this engine can do; three were literals the message declared about itself. None could go red when the engine changed. The reviewer measured it rather than arguing it: replacing the local-cause sentence with a self-declared falsehood while keeping all five substrings left `test_ltx2_video` at 44 cases / 914 assertions / SUCCESS. The case now **measures the engine first**. It renders with and without an appending conditioning item, reads `video_tokens` — the one trace field written inside the phase loop, so the only one that can observe what the loop does — and requires the grown count to exceed the plain one and both renders to return at the target frame count. Only then does it constrain the message, and the property is positional rather than lexical: every occurrence of a closed cause must sit after the `WHAT IS *NOT* THE REASON` marker, because recording a ruled-out cause is this message's own convention and has to stay possible. **R1 re-run on the repaired case**, at `f727cfd85`: `git diff --stat` 1 file, +13/-29; **BUILT** yes, `: error:` count 0; **exit 1**, 49 cases → 1 failed, 1140 assertions → 1 failed (the suite was 49 cases before the #929 merge added one). It fails on `REQUIRE(ruled_out != npos)` — the restored message has no ruled-out section at all — and would fail again on the positional check. ## The bf16 headline was only two thirds gated The aggregation dtype binds **three** roundings and only two had cases: `B * strength` (`fuse_loras.py:113`) and `deltas.add_(weight)` (`:67-68`). The matmul result's own `.to(dtype=dtype)` had none. **Measured:** widening only that one — keeping the f32 accumulator and adding the weight to it before the single store — left `test_ltx2_lora` at 13/13 and `test_ltx2_loader` at 31/31. The new case puts `acc = 1 + 2^-8` exactly on a bf16 tie and adds `w = 2^-9`, so the ported order stores 1.0 and an f32 accumulator stores 1.0078125 — one bf16 step apart in the STORED result, where the final rounding cannot absorb it. **Mutation:** `git diff --stat` 1 file, +4/-4; **BUILT** yes, `: error:` count 0; **exit 1**, `test_ltx2_lora` 14 cases → 1 failed, 2 assertions, reporting `16257 == 16256`. `test_ltx2_loader` stayed 31/31 under the same mutation, and the other 13 lora cases stayed green, which is the measurement that the hole was real. ## What lands * **The adapter reader** (`ltx2_lora.h` / `.cpp`): `.lora_A.weight` / `.lora_B.weight` pairs resolved onto the DiT contract through upstream's ComfyUI prefix strip (`sd_ops.py:135-137`), plus the file's whole `__metadata__`. * **The fusion**: `sum((B * strength) @ A)` added at load, mirroring `fuse_loras.py:99-116`. * **Every dtype arm from one hook**, placed immediately after `MaterializeDitTensor` because both quantized branches already `return vt::DType::kBF16`. On the streaming arm it runs before the device copy, so that arm's "one host buffer live at a time" invariant is unchanged. * **The surface**: `lora_path` / `lora_strength` load extras and `ltx2-gen --lora PATH [STRENGTH]`. No ABI change — both ride the existing parallel extras arrays. Load-time rather than per-request, because upstream takes the LoRAs as a `DiffusionStage.from_checkpoint` constructor argument (`ic_lora.py:104-114`). ## Two deliberate divergences, both argued rather than silent **We do not re-quantize.** Upstream's FP8 and NVFP4 rules dequantize, add, and re-quantize (`fp8_scaled_mm.py:167-189`, `nvfp4/fuse.py:13-50`) because they keep packed weights resident for their quantized kernels. This tree materializes bf16 on every arm and carries no FP8 or NVFP4 quantizer at all, so there is nothing to re-quantize into. Our fused weight skips upstream's lossy round trip and is slightly *more* precise on those two arms, at no extra bytes. **An adapter naming a module the contract lacks REFUSES.** Upstream skips it (`fuse_loras.py:135-137`) because its state dict is the whole model. Here the contract is a fixed enumerated set with unported modules already stripped, so a skip would absorb a misnamed key and an inapplicable one alike. ## `kLoraFusion` is retired, not reclassified It carried `DECLARED, NOT REQUESTABLE` — an assertion that no request field or load extra asks for LoRA fusion. One now does. #691 predicted this exact drift in its own words and records that the ledger test gates the message *text* rather than the property. The compiler caught it here, which is weaker than what #691 asks for and **does not close #691**. ## Reachability Proven on the rendered pixels, not on `last_conditioning()`: the conditioning trace is filled before the denoise loop, so it cannot see a fused weight, and a first attempt comparing it found every arm identical for that reason rather than because the LoRA did nothing. Production entry point: `vllm_video_engine_load` → `LoadVideoEngine` → `Ltx2VideoEngine::Load` → `Ltx2LoadDitFromSafetensors`. The test enters there with a `lora_path` load extra and compares rendered artifact bytes against an identical request with no adapter. ## Mutations | # | mutation | diff | BUILT | exit | result | |---|---|---|---|---|---| | M1 | accumulate the delta in f32 instead of bf16 | 1 file, +4/-4 | yes, no compile_err | 1 | 13 cases → 1 failed, 3 assertions | | M2 | **reachability**: delete `dit_options.loras.push_back` | 1 file, +2/-1 | yes, no compile_err | 1 | 44 cases → 5 failed, 8 assertions | | M3 | disable the unknown-target refusal (upstream's skip) | 1 file, +1/-1 | yes, no compile_err | 1 | 13 cases → 1 failed, 3 assertions | | M4 | ignore the adapter strength | 1 file, +1/-1 | yes, no compile_err | 1 | 13 cases → 1 failed, 6 assertions | | M5 | remove the zero-fusion refusal | 1 file, +1/-1 | **no** — `-Werror=unused-parameter` | NOT_RUN | establishes nothing; redone as M5b | | M5b | same, written to compile (`fused >= 0`) | 1 file, +1/-1 | yes, no compile_err | 1 | 31 cases → 1 failed, 2 assertions | | M6 | report the A factor under the B key | 1 file, +1/-1 | yes, no compile_err | 1 | 13 cases → 1 failed, 2 assertions | | **R1** | restore the pre-repair reference refusal verbatim | 1 file, +13/-29 | yes, no compile_err | 1 | 49 cases → 1 failed | | **F3** | widen ONLY the matmul-result rounding to f32 | 1 file, +4/-4 | yes, no compile_err | 1 | `test_ltx2_lora` 14 → 1 failed, 2 assertions; `test_ltx2_loader` 31/31 GREEN | R1 and F3 were run at `f727cfd85`, the commit before the #929 merge. Both carry to the pushed head: `git diff f727cfd e367026` is EMPTY over `ltx2_lora.cpp` and `test_ltx2_lora.cpp`, and over `ltx2_video.cpp` it changes only the refusal-counter comment and #929's own additions, leaving the reference `Fail(...)` string byte-identical. M1 to M6 were run before the review, at the counts the suites had then. Every mutation was restored byte-for-byte, proven by a clean `git status`. M5 is reported rather than dropped because a mutation that fails to build reads exactly like a passing test. ## On porting upstream's tests There are none to port. Measured at the pin with a positive control so a null result cannot be a wrong search term: `find -name 'test_*.py'` → 0, `find -type d -name 'test*'` → 0, `conftest.py` → 0, `grep -rl 'import pytest\|import unittest'` → 0, against `find -name '*.py'` → **280**. ## Records, and two merges resolved by key `origin/main` moved three times during this repair — #966, #939 and #929 — so the branch carries two merges. Both keyed records were resolved by taking main's version and reapplying this row's scoped edit. * **`docs/FEATURES.md`**: **186 of 188 keys byte-identical** to `origin/main`, the two that differ are the two this row owns, and no key is added or removed. Main's LTX-2.5 DiT cell had trimmed `IMAGE` to `IMG` and `Speed PENDING` to `PENDING` to fit `GENkf`; that trim is kept and this row's own wording carried the cost. The merged cell is **218 characters against the 220 `MAX_CELL_CHARS` limit**, and no measurement or host qualifier was touched. * **`docs/USAGE.md`**: main's served-last-frame-keyframe paragraph taken whole; the reference paragraph rewritten, because both sides' reasons for that refusal are now false, and the page records that rather than deleting it. * **`.agents/issue-index.md`**: the branch had appended a **second** `#930` row describing that blocker as open. #930 is closed, and main already carries the authoritative row, so the duplicate is dropped before it lands — the append-only rule protects rows that exist on `main`, and this one never did. The `#932` row's "blocked behind #930" is corrected, and **#975** is appended. `issue-index append-only` passes in preflight. * **`READER ANCHORS`**: both sides had rewritten the list, so neither survived the merge. Re-derived a third time with the same algorithm `test_ltx2_video` uses: `779 789 790 852 948 964 966 1044 1069 1174 1215`. * **The refusal counter stays at SIX.** #929 and this row both rewrote a refusal onto token-append, and `c7cb59fbb` falsified both while both were open in review. #929 landed leaving the counter at six, reasoning that a near-miss caught in review is not one of them; that reasoning applies here too. The comment records both near-misses rather than claiming a seventh, and a SIX and a SEVEN would have auto-merged into one of them silently. ## Gate ``` HEAD=e367026cd dirty_files=0 CONFIGURE_EXIT=0 BUILD_EXIT=0 ": error:" count=0 "No space left"=0 "BFD.*assertion"=0 control_no_space=1 control_bfd=1 control_error=1 free disk 42G ctest -N: Total Tests: 489 CTEST_N_EXIT=0 CTEST_EXIT=0 100% tests passed, 0 tests failed out of 489 loadavg 5.37 at start, 5.36 at end scripts/agent-preflight.sh: PREFLIGHT_EXIT=0 ``` The three `grep` counts carry **positive controls**, so a zero cannot be a broken instrument: the same patterns match their own sample strings and return 1. `scripts/check-doc-checkpoint.py` walks **per commit** (#573) and was run that way, armed with a positive control: `--commit b5618b3` exits **1** with the `user_usage`/`docs/USAGE.md` message, and both new commits plus the whole `origin/main..HEAD` range exit **0**. The gate on the pre-#929 tree was also green — 489/489, exit 0 — and is reported only to say that the #929 merge did not change the verdict. ## Owed * **#975** — serving the reference arms: the reference clip's own pixel path, and the stage split that gives stage 2 no adapter. Token-append is no longer part of it. * **#932** — the `conditioning_attention_strength < 1.0` / `conditioning_attention_mask` arm, and N-adapter fusion. The latter needs upstream's second rounding pattern (`addmm_` with `alpha`, `fuse_loras.py:115`), which this row refuses rather than guesses. GGUF k-quant LoRA fusion is **not applicable** rather than owed: the LTX-2.5 DiT ships FP8 and NVFP4, and no GGUF LTX DiT exists to fuse into. ## Not measured, and not implied No real-weights IC-LoRA fusion, and no render-quality or speed claim. **No GPU was used.** The fixture render moves 33 of 91169 artifact bytes, which is a reachability witness on a 2-layer reduced DiT and is not a quality result. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The LTX-2.5 phase loop was fixed at one
Ltx2VideoTokenCount(vshape, 1), and that single limitation blocked three conditioning arms. Two rows walked into it hours apart without talking to each other — #930 from the IC-LoRA port and #920 from the generated-keyframe-slots port — which is why it gets a seam of its own rather than a fix inside whichever row noticed it last.Issue: #930. Spec:
.agents/specs/ltx25-token-append.md. Campaign #644. PinLightricks/LTX-2 @ fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca, verified at the working checkout.Why this branch replaces #948
This is a history repair, not a content one. #948 carried this row through implementation and a fresh review to PASS at head
a1b1d8da4. It cannot land, and the reason is entirely in its commit ORDER.scripts/check-doc-checkpoint.pywalks a range per commit (#573). The review-repair commitb5618b305editedinclude/vllm/model_executor/models/ltx2_conditioning.handinclude/vllm/multimodal/ltx2_video.h— theuser_usageclass — while thedocs/USAGE.mdedit that pays for them sat in a sibling commit. The final tree satisfied the obligation; only the ordering failed. Measured on this branch, the checker still REDs onb5618b305in isolation and is green on both commits here, so the instrument is armed rather than merely quiet.A commit message cannot be corrected in place and this repository uses no force variant, so the branch is rebuilt with the shape the checker requires: one spec commit, then one implementation commit carrying
docs/USAGE.mdanddocs/FEATURES.mdalongside the code they describe. #678 was rebuilt as #880 and #661 as #882 for exactly this rule.The reviewed content is proven unchanged. Built on
2daa3287f—origin/mainwhen the rebuild started — this branch's head tree hash wasd937252dd06efd6a07c4affebff42153b25075a5, byte-for-byte equal toa1b1d8da4^{tree}, andgit diff a1b1d8da4 HEADwas empty.origin/mainthen advanced tof365cc299(#958), so the branch was rebased onto it; restricted to this row's ten files, the only remaining differences againsta1b1d8da4are the two lines #958 itself changed — its own appended issue-index row and its--offload-configrewrite indocs/USAGE.md. The eight code, test, spec anddocs/FEATURES.mdfiles are byte-identical. Nothing below was re-derived.The sizing in #930 is corrected, not inherited
#930 names two blocked arms. There are three: reference video, the last-frame keyframe, and generated keyframe slots. #930 predates #920 and its author asked for the correction. This spec is the record that supersedes both counts.
What was actually missing
The appending conditioning ITEMS were already ported and gated. Two halves were not.
Ltx2ExtendKeyframesMaskmirrorsextend_keyframes_mask(conditioning/mask_utils.py:76-105). Upstream's docstring makes the call an obligation of appending itself — "Every conditioning item that appends tokens must call this, otherwise the per-token marker goes out of sync with the token sequence" (:83-85) — so it lives insideAppendTokensrather than at the three call sites. BothNonebranches are mirrored, including the zero-fill onlymarked=truereaches (:100-101), so #920's arm finds it already correct.Ltx2ClearConditioningmirrorsclear_conditioning(tools.py:88-117). Two things there are not a truncation: the denoise mask comes back all ones rather than sliced (:104), because the returned state describes a finished latent, andkeyframes_maskis dropped entirely (:113).Ltx2LatentStategrows akeyframes_maskfield so the marker can be extended with the sequence it describes.Three findings that changed the design
The attention mask is not the gap, and no field is added for it. Both appending video items pass a literal
attention_mask=None(keyframe_cond.py:70,reference_video_cond.py:88),update_attention_maskreturnsNonefor that case (mask_utils.py:110-143), and the only route to a non-Nonemask isConditioningItemAttentionStrengthWrapper, whose sole application site isiclora_utils.py:169on the IC-LoRA path.combined_image_conditionings— the route this engine mirrors — never wraps. A field here would be one no ported item could populate, which is the unpassed-parameter shape.agents/reachability.mdenumerates. The refusals that cite the absent field were right to mention it and wrong to offer it as the blocker.The sigma schedule must keep reading the target count. Upstream fixes it twice over: the argument is
math.prod(latent.shape[2:])of the unpatchified target (schedulers.py:32), which by construction cannot contain appended tokens, and every pipeline computes its sigmas before a state exists (ti2vid_one_stage.py:207passes no latent;distilled.py:200-201uses frozen constants). This engine's call sits after the conditioning block, so before this change it would have re-shifted the whole trajectory the moment anything appended. It now reads atarget_tokenslocal.The trim cannot be gated on pixels alone, so it is gated on a guard. Appended tokens sit at the tail of a contiguous
[tokens, width]buffer andLtx2VideoUnpatchifytakes a bare pointer, so an un-trimmed state would unpatchify the same head bytes and render pixel-identical frames. Correct output for the wrong reason. AVT_CHECKat the pointer boundary is what turns "the head happens to be right" into "the buffer is the target grid", and it is what makes deleting the trim a RED (mutation M3).The arm lifted, and the two left alone
The last-frame keyframe arm is served, mirroring
combined_image_conditionings(helpers.py:272-308): one preprocess-and-encode shared by both arms, and a branch onframe_idxsending 0 toVideoConditionByLatentIndex(:296) and anything else toVideoConditionByKeyframeIndex(:302). Both may be supplied at once. Nothing is added toinclude/vllm.h—last_framewas already on the ABI.Reference video stays refused and its message is byte-identical. At this base, PR #938 (
row/LTX25-IC-LORA) is open and unmerged andgit log --grep '#923'is empty, so--loradoes not read the IC-LoRA scale factors and that refusal's stated cause is still true. #930's body describes the message as already rewritten onto token-append; that is not the state oforigin/main. Rewording it would have shipped a refusal naming the first of two causes — the defect a sibling row nearly shipped.Generated keyframe slots (#920, PR #929, also open) and reference audio stay refused for their own reasons.
What lands unreached, named rather than discovered
Ltx2ExtendKeyframesMask'smarked=truebranch lands with a unit driver and no production caller. The only upstream construct that passestrueisVideoGeneratedKeyframeSlots(keyframe_slots.py:121), which belongs to the generated-keyframe-slots arm and not to this row. Every appending item ported here passesfalse.It is landed rather than stubbed because building half of
extend_keyframes_maskis the worse option: the twoNonebranches are not symmetric, and a port that mirrors one and guesses the other gets #920's arm silently wrong. Mutation M5 shows the branch is not inert — appending withmarked=trueinstead offalseis RED.Owner of the wiring: row
LTX25-GENERATED-KEYFRAMES, issue #920. Listed under## Owedin the spec. Declared here per.agents/reachability.md## Landing a slice that is not reached yet.Evidence
RED first, on the pre-row engine with the tests in place:
test_ltx2_videoFAILED, the new case throwing the last-frame refusal verbatim. That run also printedassertions: 825 | 825 passed | 0 failedbesideStatus: FAILURE!— the thrown-case trap, which is why the exit code is the authority throughout.The witness is on rendered artifact bytes with a no-op control, because
Ltx2ConditioningTraceis filled before denoise and cannot observe the loop. A keyframe render differs from a no-keyframe render in 59773 of 91169 bytes; two different keyframes differ from each other in 334; the same request twice is byte-identical. The control is what separates "the append reached the maths" from "the instrument is blind" — a sibling row's first attempt found every arm identical including its control, and without that control a blind instrument reads as a real-but-subtle effect, which is the direction that ships.Two new trace fields,
video_tokensandschedule_tokens, are written inside the phase loop for that same reason, and the sigma claim is checked as a relation between two measured values rather than against a literal.Upstream ships no tests —
find /home/mudler/_git/LTX-2 -name 'test_*.py'returns 0 across the whole repository at the pin — so nothing was ported and every case is written against an upstream anchor instead.Review repairs
A fresh review returned FAIL on six findings. The seam itself survived: every upstream design decision checked out and the pixel witness reproduced exactly. Two claimed guarantees survived mutation and are now gated.
The all-ones restore was not gated.
Ltx2CreateVideoLatentStatealready sets every target-range mask value to1.0, and the append writes1 - strengthonly at the tail, which the trim drops under either implementation — so "restore all ones" and "slice" produced identical bytes over the range the test walked, and mutation M6 (slice instead of restore) was GREEN. The case now conditions token 0 inside the target withLtx2ConditionVideoByLatentIndexbefore the append, which writes1 - strengthatstart .. start + count(latent_cond.py:41). A slice then leaves0.4at token 0. M6 is RED, and aREQUIREonmask.front()before the trim asserts the instrument is armed rather than assuming it.Nothing gated that the keyframe lands on the last frame. Mutation M10 (
frame_idxfromframes - 1to0) was GREEN: both renders still differed from the no-op control and from each other, and the token count was identical, because a keyframe pinned to the first frame appends exactly as many tokens as one pinned to the last. The engine now asserts the first appended token's temporal position, recomputed fromframesandfpsrather than read back from the argument, so the two are independent expressions (keyframe_cond.py:52-58). M10 is RED.origin/mainadvanced toc2019b0e3(#935) during review and the branch stopped merging. Three conflicts, two with a trap:docs/FEATURES.md's LTX-2.5 cell is atMAX_CELL_CHARSexactly, so the obvious merge lands at 226 and REDs — resolved by keeping main's complete cell and trimming only this row's own wording, to 220. And both sides had rewritten theREADER ANCHORSline, so neither value was correct afterwards; it was re-derived a third time from the merged file.Every upstream
file:linein this change was re-derived at the final tree by a needle taken from the claim, never read out of the cited span, and the load-bearing needles were checked for uniqueness in their files. The sweep covers 119 citations in this row's added lines — 108 explicit and 11 bare:NNcontinuations — and corrected 13.Four of those were the review's:
tools.py:103and:112in theLtx2ClearConditioningdeclaration and in the spec, andschedulers.py:38-39in four places including the append-only issue-index row, which is fixed first and SHA-anchored because it could not be corrected after landing.Three were not, and two of those three are anchors written during this repair and caught by the sweep over its own output:
ltx2_recipes.py:125-158names a file that exists neither upstream nor in this tree. The distilled recipe's frozen sigmas aredistilled.py:200-201overutils/constants.py:17-23.(:100-101)resolves againsttools.py, the nearest file named above it, whose:100-101is a real and unrelated statement. It meansmask_utils.py. The two bare continuations inLtx2ClearConditioning's declaration are spelled with their file for the same reason: the nearest name above them isblocks.py, andblocks.py:104and:113are both real imports.latent_cond.py:36-38, written here for the M6 repair, isstop_tokenand aclone(). The1 - strengthwrite is:41.One anchor is left alone deliberately.
latent_cond.py:38-39is cited at eight pre-existing sites across specs, golden scripts and other rows' tests for "onlyclean_latentanddenoise_maskare written". Those writes are:40-41;:38-39is aclone()and a blank line. The drift predates this row and the sites belong to other rows, so correcting them here would be scope this PR did not claim. It is named rather than left for the next sweep to rediscover.Gate on the rebuilt branch
Rerun in full rather than inherited, on the rebased head
fd9948f14, CPU-only Release,-j6build andctest -j4:CONFIGURE_EXIT=0,BUILD_EXIT=0,: error:count 0,No space left0,BFD assertion0 (all three greps positive-controlled against a synthetic line),ctest -N485,CTEST_EXIT=0,100% tests passed, 0 tests failed out of 485. Load average 4.94 at start and 11.02 at end; 46 G free.Row suites, run directly for their assertion counts:
test_ltx2_video44/44 cases, 1024/1024 assertions;test_ltx2_vae39/39, 3092/3092;test_ltx243/43, 4388/4388;test_ltx2_device18/18, 546/546.M6 and M10 were rerun on this branch rather than carried over. M6 —
Ltx2ClearConditioningslices instead of restoring —git diff --statone file1 +/1 -, BUILT with 0 compile errors, exit 1:38 passed | 1 failed, fourCHECK( 0.4 == 1 )at mask tokens 0-3. M10 —frame_idxframes - 1to0— one file1 +/1 -, BUILT with 0 compile errors, exit 1:42 passed | 2 failed, both cases THREW the temporal-positionVT_CHECK(0.333333wanted,0.000000got). M10 printsassertions: 923 | 923 passed | 0 failedbesideStatus: FAILURE!, which is the thrown-case trap; the exit code is the authority. The tree was restored byte-for-byte after each —git write-treeback tod937252dd0— and rebuilt before remeasuring.READER ANCHORSwere re-derived at the final tree and are unchanged:756 811 907 923 925 1003 1028 1133 1174. Thedocs/FEATURES.mdLTX-2.5 support cell measures 220 characters, exactlyMAX_CELL_CHARS, with main's, A2V WAVand every host and measurement qualifier intact.Checkers, exit codes captured directly:
check-doc-checkpoint0 over the range and 0 on each commit individually,check-commit-trailers0,check-commit-style0,check-issue-index-append-only0,check-agent-record0,check-public-doc-tables0.FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]