Skip to content

docs(FIX-OFFLOAD-DOCS-957): weight offload is refused at startup, not accepted and inert (#957) - #958

Merged
localai-bot merged 1 commit into
mainfrom
row/FIX-OFFLOAD-DOCS-957
Aug 15, 2026
Merged

docs(FIX-OFFLOAD-DOCS-957): weight offload is refused at startup, not accepted and inert (#957)#958
localai-bot merged 1 commit into
mainfrom
row/FIX-OFFLOAD-DOCS-957

Conversation

@localai-bot

Copy link
Copy Markdown
Collaborator

docs(FIX-OFFLOAD-DOCS-957): weight offload is refused at startup, not accepted and inert

Closes #957. No code moves.

4a183b731 (#887) turned a configured weight offload from ACCEPTED-AND-INERT
into a hard startup refusal, and neither public document followed. Both still
described the behaviour that commit replaced, so a reader who followed them
would add --offload-config expecting a no-op and get a server that does not
start.

What the engine actually does

RefuseUnsupportedWeightOffload (weight_offloader.cpp:72-83) throws when
offloading is enabled and the resolved architecture does not declare support. It
is called unconditionally from the load path (model_loader.cpp:1410-1414),
right after ModelRegistry::Resolve and BEFORE any weight I/O.

ModelFactory::supports_weight_offload defaults to false and NO model in the
tree sets it, so EVERY architecture is refused today. The tree asserts this
itself: test_weight_offloader.cpp:376-379 counts the declaring models and
carries the message "a model now declares supports_weight_offload; update this".
Negative control on the grep: a misspelled needle returns 0, so the empty set is
not a wrong search term.

A second refusal from that commit was also undocumented —
VerifyWeightOffloadWasConsulted throws after load when a model declares
support and then consults the offloader about zero weights, reported as a defect
in that loader rather than a configuration error. The old refusal list covered
config validation only.

Checked, not assumed

The example error names Qwen3MoeForCausalLM, which is a real registered
architecture (qwen3_moe_registry.cpp:181). An invented name in an error
example is indistinguishable from a real one to a reader. The "only when
enabled" claim is weight_offloader.cpp:75. The quoted message is transcribed
from the throw.

docs/STATUS.md needed no change: it already lists weight offload under what is
owed and never claimed the inert behaviour.

THIS DOES NOT MAKE documentation-checkpoint GREEN

check-doc-checkpoint.py evaluates each commit against its OWN parent, so no
later commit can satisfy an earlier one, and there is no waiver mechanism
(:256 says exempting named paths was deliberately avoided). Measured:
docs/FEATURES.md/docs/USAGE.md changed in 66 commits since 73d217db, which
still reports ERROR. The 28-commit backlog is closed only by advancing the base
or changing the checker, and this row takes neither decision.

The other 27 owed nothing

Audited all 28. The 19 MODEL-MM-indextts2 commits are intermediate waves whose
campaign documented itself at f374ab8ed once the pipeline rendered end to end,
and every one is an ancestor of it. The 3 ENG-EXPERT-STREAM commits are
deliberately unwired staged slices that docs/FEATURES.md must NOT claim —
#918's own body says "Nothing wired to a loader". Four earlier
ENG-WEIGHT-OFFLOAD commits predate 62406c30e, which documented them.
fba312c67 is a record/measurement commit shipping no user surface.

A staged campaign that documents itself once, when the capability becomes
reachable, is the CORRECT shape. The gate flags every wave that is not that
commit, so a large flagged count is expected and is not by itself evidence of
stale docs. Only a commit landing user-visible behaviour AFTER its campaign's
docs can make a document wrong. Exactly one of the 28 did.

Spec: .agents/specs/offload-docs-refusal.md.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

… accepted and inert

Closes #957. No code moves.

`4a183b731` (#887) turned a configured weight offload from ACCEPTED-AND-INERT
into a hard startup refusal, and neither public document followed. Both still
described the behaviour that commit replaced, so a reader who followed them
would add `--offload-config` expecting a no-op and get a server that does not
start.

## What the engine actually does

`RefuseUnsupportedWeightOffload` (`weight_offloader.cpp:72-83`) throws when
offloading is enabled and the resolved architecture does not declare support. It
is called unconditionally from the load path (`model_loader.cpp:1410-1414`),
right after `ModelRegistry::Resolve` and BEFORE any weight I/O.

`ModelFactory::supports_weight_offload` defaults to false and NO model in the
tree sets it, so EVERY architecture is refused today. The tree asserts this
itself: `test_weight_offloader.cpp:376-379` counts the declaring models and
carries the message "a model now declares supports_weight_offload; update this".
Negative control on the grep: a misspelled needle returns 0, so the empty set is
not a wrong search term.

A second refusal from that commit was also undocumented —
`VerifyWeightOffloadWasConsulted` throws after load when a model declares
support and then consults the offloader about zero weights, reported as a defect
in that loader rather than a configuration error. The old refusal list covered
config validation only.

## Checked, not assumed

The example error names `Qwen3MoeForCausalLM`, which is a real registered
architecture (`qwen3_moe_registry.cpp:181`). An invented name in an error
example is indistinguishable from a real one to a reader. The "only when
enabled" claim is `weight_offloader.cpp:75`. The quoted message is transcribed
from the throw.

`docs/STATUS.md` needed no change: it already lists weight offload under what is
owed and never claimed the inert behaviour.

## THIS DOES NOT MAKE documentation-checkpoint GREEN

`check-doc-checkpoint.py` evaluates each commit against its OWN parent, so no
later commit can satisfy an earlier one, and there is no waiver mechanism
(`:256` says exempting named paths was deliberately avoided). Measured:
`docs/FEATURES.md`/`docs/USAGE.md` changed in 66 commits since `73d217db`, which
still reports ERROR. The 28-commit backlog is closed only by advancing the base
or changing the checker, and this row takes neither decision.

## The other 27 owed nothing

Audited all 28. The 19 `MODEL-MM-indextts2` commits are intermediate waves whose
campaign documented itself at `f374ab8ed` once the pipeline rendered end to end,
and every one is an ancestor of it. The 3 `ENG-EXPERT-STREAM` commits are
deliberately unwired staged slices that `docs/FEATURES.md` must NOT claim —
#918's own body says "Nothing wired to a loader". Four earlier
`ENG-WEIGHT-OFFLOAD` commits predate `62406c30e`, which documented them.
`fba312c67` is a record/measurement commit shipping no user surface.

A staged campaign that documents itself once, when the capability becomes
reachable, is the CORRECT shape. The gate flags every wave that is not that
commit, so a large flagged count is expected and is not by itself evidence of
stale docs. Only a commit landing user-visible behaviour AFTER its campaign's
docs can make a document wrong. Exactly one of the 28 did.

Spec: `.agents/specs/offload-docs-refusal.md`.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
@localai-bot
localai-bot merged commit f365cc2 into main Aug 15, 2026
19 of 25 checks passed
@localai-bot
localai-bot deleted the row/FIX-OFFLOAD-DOCS-957 branch August 15, 2026 21:24
localai-bot pushed a commit that referenced this pull request Aug 15, 2026
FOLLOWING_AGENTS_PROTOCOL

Keeps the branch current before review. No content of this row changes; the
incoming commit is the weight-offload docs fix (#958).

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]
localai-bot added a commit that referenced this pull request Aug 16, 2026
…rim it back (#930) (#964)

The LTX-2.5 phase loop was fixed at one `Ltx2VideoTokenCount(vshape, 1)`, and that single limitation blocked three conditioning arms. Two rows walked into it hours apart without talking to each other — [#930](#930) from the IC-LoRA port and [#920](#920) from the generated-keyframe-slots port — which is why it gets a seam of its own rather than a fix inside whichever row noticed it last.

Issue: [#930](#930). Spec: [`.agents/specs/ltx25-token-append.md`](.agents/specs/ltx25-token-append.md). Campaign [#644](#644). Pin `Lightricks/LTX-2 @ fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca`, verified at the working checkout.

## Why this branch replaces #948

**This is a history repair, not a content one.** [#948](#948) carried this row through implementation and a fresh review to PASS at head `a1b1d8da4`. It cannot land, and the reason is entirely in its commit ORDER.

`scripts/check-doc-checkpoint.py` walks a range **per commit** ([#573](#573)). The review-repair commit `b5618b305` edited `include/vllm/model_executor/models/ltx2_conditioning.h` and `include/vllm/multimodal/ltx2_video.h` — the `user_usage` class — while the `docs/USAGE.md` edit that pays for them sat in a sibling commit. The final tree satisfied the obligation; only the ordering failed. Measured on this branch, the checker still REDs on `b5618b305` in isolation and is green on both commits here, so the instrument is armed rather than merely quiet.

A commit message cannot be corrected in place and this repository uses no force variant, so the branch is rebuilt with the shape the checker requires: **one spec commit, then one implementation commit** carrying `docs/USAGE.md` and `docs/FEATURES.md` alongside the code they describe. #678 was rebuilt as #880 and #661 as #882 for exactly this rule.

**The reviewed content is proven unchanged.** Built on `2daa3287f` — `origin/main` when the rebuild started — this branch's head tree hash was `d937252dd06efd6a07c4affebff42153b25075a5`, byte-for-byte equal to `a1b1d8da4^{tree}`, and `git diff a1b1d8d HEAD` was empty. `origin/main` then advanced to `f365cc299` (#958), so the branch was rebased onto it; restricted to this row's ten files, the only remaining differences against `a1b1d8da4` are the two lines #958 itself changed — its own appended issue-index row and its `--offload-config` rewrite in `docs/USAGE.md`. The eight code, test, spec and `docs/FEATURES.md` files are byte-identical. Nothing below was re-derived.

## The sizing in #930 is corrected, not inherited

#930 names two blocked arms. There are three: reference video, the last-frame keyframe, and generated keyframe slots. #930 predates #920 and its author asked for the correction. This spec is the record that supersedes both counts.

## What was actually missing

The appending conditioning ITEMS were already ported and gated. Two halves were not.

`Ltx2ExtendKeyframesMask` mirrors `extend_keyframes_mask` (`conditioning/mask_utils.py:76-105`). Upstream's docstring makes the call an obligation of appending itself — *"Every conditioning item that appends tokens must call this, otherwise the per-token marker goes out of sync with the token sequence"* (`:83-85`) — so it lives inside `AppendTokens` rather than at the three call sites. Both `None` branches are mirrored, including the zero-fill only `marked=true` reaches (`:100-101`), so #920's arm finds it already correct.

`Ltx2ClearConditioning` mirrors `clear_conditioning` (`tools.py:88-117`). Two things there are not a truncation: the denoise mask comes back **all ones** rather than sliced (`:104`), because the returned state describes a finished latent, and `keyframes_mask` is dropped entirely (`:113`).

`Ltx2LatentState` grows a `keyframes_mask` field so the marker can be extended with the sequence it describes.

## Three findings that changed the design

**The attention mask is not the gap, and no field is added for it.** Both appending video items pass a literal `attention_mask=None` (`keyframe_cond.py:70`, `reference_video_cond.py:88`), `update_attention_mask` returns `None` for that case (`mask_utils.py:110-143`), and the only route to a non-`None` mask is `ConditioningItemAttentionStrengthWrapper`, whose sole application site is `iclora_utils.py:169` on the IC-LoRA path. `combined_image_conditionings` — the route this engine mirrors — never wraps. A field here would be one no ported item could populate, which is the unpassed-parameter shape [`.agents/reachability.md`](.agents/reachability.md) enumerates. The refusals that cite the absent field were right to mention it and wrong to offer it as the blocker.

**The sigma schedule must keep reading the target count.** Upstream fixes it twice over: the argument is `math.prod(latent.shape[2:])` of the *unpatchified* target (`schedulers.py:32`), which by construction cannot contain appended tokens, and every pipeline computes its sigmas before a state exists (`ti2vid_one_stage.py:207` passes no latent; `distilled.py:200-201` uses frozen constants). This engine's call sits **after** the conditioning block, so before this change it would have re-shifted the whole trajectory the moment anything appended. It now reads a `target_tokens` local.

**The trim cannot be gated on pixels alone, so it is gated on a guard.** Appended tokens sit at the tail of a contiguous `[tokens, width]` buffer and `Ltx2VideoUnpatchify` takes a bare pointer, so an un-trimmed state would unpatchify the same head bytes and render pixel-identical frames. Correct output for the wrong reason. A `VT_CHECK` at the pointer boundary is what turns "the head happens to be right" into "the buffer *is* the target grid", and it is what makes deleting the trim a RED (mutation M3).

## The arm lifted, and the two left alone

The **last-frame keyframe** arm is served, mirroring `combined_image_conditionings` (`helpers.py:272-308`): one preprocess-and-encode shared by both arms, and a branch on `frame_idx` sending 0 to `VideoConditionByLatentIndex` (`:296`) and anything else to `VideoConditionByKeyframeIndex` (`:302`). Both may be supplied at once. Nothing is added to `include/vllm.h` — `last_frame` was already on the ABI.

**Reference video stays refused and its message is byte-identical.** At this base, PR #938 (`row/LTX25-IC-LORA`) is open and unmerged and `git log --grep '#923'` is empty, so `--lora` does not read the IC-LoRA scale factors and that refusal's stated cause is still true. #930's body describes the message as already rewritten onto token-append; **that is not the state of `origin/main`.** Rewording it would have shipped a refusal naming the first of two causes — the defect a sibling row nearly shipped.

**Generated keyframe slots** (#920, PR #929, also open) and **reference audio** stay refused for their own reasons.

## What lands unreached, named rather than discovered

`Ltx2ExtendKeyframesMask`'s **`marked=true` branch lands with a unit driver and no production caller.** The only upstream construct that passes `true` is `VideoGeneratedKeyframeSlots` (`keyframe_slots.py:121`), which belongs to the generated-keyframe-slots arm and not to this row. Every appending item ported here passes `false`.

It is landed rather than stubbed because building half of `extend_keyframes_mask` is the worse option: the two `None` branches are not symmetric, and a port that mirrors one and guesses the other gets #920's arm silently wrong. Mutation M5 shows the branch is not inert — appending with `marked=true` instead of `false` is RED.

Owner of the wiring: row `LTX25-GENERATED-KEYFRAMES`, issue [#920](#920). Listed under `## Owed` in the spec. Declared here per [`.agents/reachability.md`](.agents/reachability.md) `## Landing a slice that is not reached yet`.

## Evidence

RED first, on the pre-row engine with the tests in place: `test_ltx2_video` FAILED, the new case throwing the last-frame refusal verbatim. That run also printed `assertions: 825 | 825 passed | 0 failed` beside `Status: FAILURE!` — the thrown-case trap, which is why the exit code is the authority throughout.

The witness is on **rendered artifact bytes with a no-op control**, because `Ltx2ConditioningTrace` is filled before denoise and cannot observe the loop. A keyframe render differs from a no-keyframe render in **59773 of 91169** bytes; two different keyframes differ from each other in **334**; the same request twice is byte-identical. The control is what separates "the append reached the maths" from "the instrument is blind" — a sibling row's first attempt found every arm identical *including* its control, and without that control a blind instrument reads as a real-but-subtle effect, which is the direction that ships.

Two new trace fields, `video_tokens` and `schedule_tokens`, are written *inside* the phase loop for that same reason, and the sigma claim is checked as a relation between two measured values rather than against a literal.

Upstream ships **no tests** — `find /home/mudler/_git/LTX-2 -name 'test_*.py'` returns 0 across the whole repository at the pin — so nothing was ported and every case is written against an upstream anchor instead.

## Review repairs

A fresh review returned FAIL on six findings. The seam itself survived: every upstream design decision checked out and the pixel witness reproduced exactly. Two claimed guarantees survived mutation and are now gated.

**The all-ones restore was not gated.** `Ltx2CreateVideoLatentState` already sets every target-range mask value to `1.0`, and the append writes `1 - strength` only at the tail, which the trim drops under either implementation — so "restore all ones" and "slice" produced identical bytes over the range the test walked, and mutation M6 (slice instead of restore) was **GREEN**. The case now conditions token 0 *inside* the target with `Ltx2ConditionVideoByLatentIndex` before the append, which writes `1 - strength` at `start .. start + count` (`latent_cond.py:41`). A slice then leaves `0.4` at token 0. M6 is **RED**, and a `REQUIRE` on `mask.front()` before the trim asserts the instrument is armed rather than assuming it.

**Nothing gated that the keyframe lands on the last frame.** Mutation M10 (`frame_idx` from `frames - 1` to `0`) was **GREEN**: both renders still differed from the no-op control and from each other, and the token count was identical, because a keyframe pinned to the first frame appends exactly as many tokens as one pinned to the last. The engine now asserts the first appended token's temporal position, recomputed from `frames` and `fps` rather than read back from the argument, so the two are independent expressions (`keyframe_cond.py:52-58`). M10 is **RED**.

`origin/main` advanced to `c2019b0e3` (#935) during review and the branch stopped merging. Three conflicts, two with a trap: `docs/FEATURES.md`'s LTX-2.5 cell is at `MAX_CELL_CHARS` exactly, so the obvious merge lands at 226 and REDs — resolved by keeping main's complete cell and trimming only this row's own wording, to 220. And both sides had rewritten the `READER ANCHORS` line, so **neither** value was correct afterwards; it was re-derived a third time from the merged file.

Every upstream `file:line` in this change was re-derived at the final tree by a needle taken from the **claim**, never read out of the cited span, and the load-bearing needles were checked for uniqueness in their files. The sweep covers **119** citations in this row's added lines — 108 explicit and 11 bare `:NN` continuations — and corrected **13**.

Four of those were the review's: `tools.py:103` and `:112` in the `Ltx2ClearConditioning` declaration and in the spec, and `schedulers.py:38-39` in four places including the append-only issue-index row, which is fixed first and SHA-anchored because it could not be corrected after landing.

Three were not, and two of those three are anchors written *during this repair* and caught by the sweep over its own output:

- `ltx2_recipes.py:125-158` names a file that exists neither upstream nor in this tree. The distilled recipe's frozen sigmas are `distilled.py:200-201` over `utils/constants.py:17-23`.
- A bare `(:100-101)` resolves against `tools.py`, the nearest file named above it, whose `:100-101` is a real and unrelated statement. It means `mask_utils.py`. The two bare continuations in `Ltx2ClearConditioning`'s declaration are spelled with their file for the same reason: the nearest name above them is `blocks.py`, and `blocks.py:104` and `:113` are both real imports.
- `latent_cond.py:36-38`, written here for the M6 repair, is `stop_token` and a `clone()`. The `1 - strength` write is `:41`.

**One anchor is left alone deliberately.** `latent_cond.py:38-39` is cited at eight pre-existing sites across specs, golden scripts and other rows' tests for "only `clean_latent` and `denoise_mask` are written". Those writes are `:40-41`; `:38-39` is a `clone()` and a blank line. The drift predates this row and the sites belong to other rows, so correcting them here would be scope this PR did not claim. It is named rather than left for the next sweep to rediscover.

## Gate on the rebuilt branch

Rerun in full rather than inherited, on the rebased head `fd9948f14`, CPU-only Release, `-j6` build and `ctest -j4`:

`CONFIGURE_EXIT=0`, `BUILD_EXIT=0`, `: error:` count **0**, `No space left` **0**, `BFD assertion` **0** (all three greps positive-controlled against a synthetic line), `ctest -N` **485**, `CTEST_EXIT=0`, `100% tests passed, 0 tests failed out of 485`. Load average 4.94 at start and 11.02 at end; 46 G free.

Row suites, run directly for their assertion counts: `test_ltx2_video` 44/44 cases, 1024/1024 assertions; `test_ltx2_vae` 39/39, 3092/3092; `test_ltx2` 43/43, 4388/4388; `test_ltx2_device` 18/18, 546/546.

M6 and M10 were rerun on this branch rather than carried over. **M6** — `Ltx2ClearConditioning` slices instead of restoring — `git diff --stat` one file `1 +/1 -`, BUILT with 0 compile errors, exit **1**: `38 passed | 1 failed`, four `CHECK( 0.4 == 1 )` at mask tokens 0-3. **M10** — `frame_idx` `frames - 1` to `0` — one file `1 +/1 -`, BUILT with 0 compile errors, exit **1**: `42 passed | 2 failed`, both cases THREW the temporal-position `VT_CHECK` (`0.333333` wanted, `0.000000` got). M10 prints `assertions: 923 | 923 passed | 0 failed` beside `Status: FAILURE!`, which is the thrown-case trap; the exit code is the authority. The tree was restored byte-for-byte after each — `git write-tree` back to `d937252dd0` — and rebuilt before remeasuring.

`READER ANCHORS` were re-derived at the final tree and are unchanged: `756 811 907 923 925 1003 1028 1133 1174`. The `docs/FEATURES.md` LTX-2.5 support cell measures **220** characters, exactly `MAX_CELL_CHARS`, with main's `, A2V WAV` and every host and measurement qualifier intact.

Checkers, exit codes captured directly: `check-doc-checkpoint` **0** over the range and **0** on each commit individually, `check-commit-trailers` **0**, `check-commit-style` **0**, `check-issue-index-append-only` **0**, `check-agent-record` **0**, `check-public-doc-tables` **0**.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs: weight offload is REFUSED at startup for every architecture since #887, but both public documents still say it is accepted and inert

2 participants