mudler · localai-bot · Aug 15, 2026 · Aug 15, 2026
diff --git a/.agents/issue-index.md b/.agents/issue-index.md
@@ -251,3 +251,4 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#874](https://github.com/mudler/vllm.cpp/issues/874) | — | `windows-msvc-cpu`/`windows-msvc-vulkan` still start on a CLOSED pull request: `check-release-workflow.py::validate_pr_ci` compares their whole job mapping for equality, so neither an `if:` clause nor a `needs:` guard can be added. Listed under `## Owed` in [`ci-concurrency.md`](specs/ci-concurrency.md) | bug |
 | [#949](https://github.com/mudler/vllm.cpp/issues/949) | — | Nothing in the tree refuses a borrowed `vt::Tensor` that outlives the object owning its storage, and the ONLY instrument that catches one is `sanitize-cpu`, which is `continue-on-error` — that is how [#904](https://github.com/mudler/vllm.cpp/issues/904) landed. Measured in the #936 review rather than argued: with the #904 fix reverted, a plain Release build with no sanitizer runs the case 18/18 passed, 546 assertions, `rc=0`, because `dtype` lives in the `vt::Tensor` struct and not in the freed buffer, so no ordinary gate can see the dangling read. Three remedies are open and none is foregone: promote the lane once it has a `main` baseline, add a test that fails without a sanitizer, or reject the pattern statically — a prototype detector for a member access chained onto a call returning an owning type by value swept 1777 files with no hit but the defect. Anchors: the owning deleter `src/vllm/model_executor/models/ltx2_device.cpp:1088 @ 800dd082f`, the read `src/vt/cpu/cpu_layernorm.cpp:33 @ 800dd082f`. Listed under `## Owed` in [`ltx2-device-staged-view-uaf.md`](specs/ltx2-device-staged-view-uaf.md) | bug |
 | [#933](https://github.com/mudler/vllm.cpp/issues/933) | `ENG-EXPERT-STREAM` | Measure gateability of the `llama-cpp-unsloth` oracle by BUILDING it and RUNNING `Qwen3.8-2.4T-A95B UD-Q1_0` on it. The oracle is pinned at `36fe8e1cc` (branch `iq1-narrow`) and records `gateable = no`, because the IQ1_XXXS port is grounded in the fork's SOURCE, read and cited, which is weaker than a running comparison. It is the only place ggml type 66 is defined: the vllm.cpp pin `237ad9b96` ends at `Q1_0 = 41` and `ggml-org` master `ad1de39e0` at `Q2_0 = 42`, while type 66 carries 96.92 % of that checkpoint's parameters. Running it needs the full 370 GiB checkpoint and, per Unsloth's documentation, at least 450 GB of RAM. Until then the ported arm has no running oracle, which is what `gateable = no` makes visible | task |
+| [#957](https://github.com/mudler/vllm.cpp/issues/957) | `FIX-OFFLOAD-DOCS-957` | `4a183b731` (#887) turned a configured weight offload from ACCEPTED-AND-INERT into a hard startup refusal, and neither public document followed. `docs/WEIGHT-OFFLOAD.md` still said "a budget you set is accepted, reported, and does not free memory" and `docs/USAGE.md:1473` still said "Accepted and inert today", while `RefuseUnsupportedWeightOffload` (`src/vllm/model_executor/weight_offloader.cpp:72-83` @ 2daa3287f) throws from the load path (`src/vllm/entrypoints/model_loader.cpp:1410-1414` @ 2daa3287f) before any weight I/O. `ModelFactory::supports_weight_offload` defaults false and NO model sets it, so every architecture is refused; `tests/vllm/model_executor/test_weight_offloader.cpp:376-379` @ 2daa3287f asserts that count itself. Found auditing the 28 commits `documentation-checkpoint` flags: 27 needed nothing | bug |
diff --git a/.agents/specs/offload-docs-refusal.md b/.agents/specs/offload-docs-refusal.md
@@ -0,0 +1,151 @@
+# The docs describe a weight offload that is accepted and inert; the engine refuses it
+
+Row: `FIX-OFFLOAD-DOCS-957`
+Issue: [#957](https://github.com/mudler/vllm.cpp/issues/957)
+Introduced by: `4a183b731` (#887, `ENG-WEIGHT-OFFLOAD` W2c)
+Baseline: `origin/main` @ `2daa3287f`
+
+## 1. Scope
+
+Correct `docs/WEIGHT-OFFLOAD.md` and `docs/USAGE.md` so they describe what the
+engine does today: refuse an enabled weight offload at startup, on every
+architecture, naming the architecture.
+
+**Out of scope:** wiring a loader (that is `ENG-WEIGHT-OFFLOAD` W2c, issue
+[#797](https://github.com/mudler/vllm.cpp/issues/797)), and changing any
+refusal. This row moves no code. It also does not attempt to make
+`documentation-checkpoint` green — §5 explains why nothing can.
+
+## 2. What the documents said, and what the engine does
+
+Both documents describe the behaviour `4a183b731` replaced.
+
+`docs/WEIGHT-OFFLOAD.md`, the callout and the summary:
+
+> A budget you set is accepted, reported, and does not free memory.
+
+> So the honest summary is: you can configure weight offload, the engine tells
+> you what it resolved, and your device memory does not change. The engine
+> prints one line at startup saying so [...]
+
+`docs/USAGE.md:1473`:
+
+> **Accepted and inert today: no weight moves yet**
+
+What actually happens, at `2daa3287f`:
+
+`RefuseUnsupportedWeightOffload`
+(`src/vllm/model_executor/weight_offloader.cpp:72-83`) throws
+`std::invalid_argument` when offloading is enabled and the resolved
+architecture's factory does not set `supports_weight_offload`. It is called
+unconditionally from the load path
+(`src/vllm/entrypoints/model_loader.cpp:1410-1414`), immediately after
+`ModelRegistry::Resolve(config)` and **before any weight I/O** — so nothing is
+read from disk before the operator is told.
+
+`ModelFactory::supports_weight_offload` defaults to `false`
+(`include/vllm/model_executor/models/model_registry.h:371`), and **no model in
+the tree sets it to `true`**. Measured by grepping `src/` and `include/` for the
+identifier: every hit is the declaration, the two guard parameters, or the
+`model_loader.cpp` read. A deliberately misspelled needle returns 0, so the
+search term is not the reason the set is empty. The tree asserts the same fact
+independently — `tests/vllm/model_executor/test_weight_offloader.cpp:376-379`
+counts the declaring models and carries the message "a model now declares
+supports_weight_offload; update this".
+
+So the practical statement is the one neither document made: **enabling weight
+offload fails startup, for every model.** A reader following the old text would
+add `--offload-config` expecting a no-op and get a server that does not start.
+
+A second refusal from the same commit was also undocumented.
+`VerifyWeightOffloadWasConsulted` (`weight_offloader.cpp:86-97`) throws after
+load when a model declares support and then asks the offloader about zero
+weights, and reports it as a defect in that loader rather than a configuration
+error. The document's refusal list covered only config validation — malformed
+document, unknown backend, wrong type, negative budget, `offload_num_in_group`
+above `offload_group_size`, `offload_prefetch_step` below 1.
+
+## 3. The change
+
+`docs/WEIGHT-OFFLOAD.md`:
+
+- the callout leads with the startup failure instead of "accepted, reported, and
+  does not free memory";
+- the *What works today* table gains a row for the model gate, so the table
+  answers "can I run with this on?" rather than only "what is built?";
+- the summary says you can write and validate a configuration and cannot yet run
+  with one enabled;
+- a new subsection under *What the engine refuses* carries the real message text
+  and the post-load consistency check.
+
+`docs/USAGE.md`: the `--offload-config` row says enabling it fails startup on
+every model, and that a config which leaves offloading disabled still parses and
+reports.
+
+Both keep the existing point that on unified memory such as GB10 offload cannot
+help regardless, because host and device share one pool.
+
+`docs/STATUS.md` needed no change: it already lists weight offload under what is
+owed and never claimed the inert behaviour.
+
+## 4. Evidence
+
+No code moves, so the gate is the accuracy of the text against the tree.
+
+| Claim in the new text | How it was checked at `2daa3287f` |
+|---|---|
+| the refusal is on the load path, before weight I/O | read `model_loader.cpp:1403-1415`; the call sits between `ModelRegistry::Resolve` and `LoadShards` |
+| it fires only when offloading is enabled | `weight_offloader.cpp:75` — `if (!config.is_offloading_enabled()) return;` |
+| the quoted message is the real one | transcribed from the `throw` at `weight_offloader.cpp:77-83` |
+| `Qwen3MoeForCausalLM` is a real architecture string | `REGISTER_VLLM_MODEL(qwen3_moe, "Qwen3MoeForCausalLM", ...)` at `qwen3_moe_registry.cpp:181` — an invented name in an error example would be indistinguishable from a real one to a reader |
+| no model declares support | grep over `src/` and `include/`; all hits are declaration, parameters, or the loader read. Negative control: a misspelled needle returns 0 |
+| the post-load check exists and is worded as a loader defect | `weight_offloader.cpp:86-97` |
+
+## 5. This does not make `documentation-checkpoint` green, and nothing can
+
+`scripts/check-doc-checkpoint.py` walks the range one commit at a time and
+evaluates each commit's **own** paths against its **own** parent (`main()`, the
+`--base`/`--head` branch, via `commit_paths(commit)`). No later commit can
+satisfy an earlier one, and there is no waiver or exemption mechanism — the file
+says at `:256` that exempting named paths was deliberately avoided.
+
+Measured rather than reasoned: `docs/FEATURES.md` and `docs/USAGE.md` have
+changed in **66 commits** since `73d217db` landed, and
+`check-doc-checkpoint.py --commit 73d217db` still reports ERROR.
+
+The 28 flagged commits are immutable on `main`, which is never rewritten. So the
+backlog is closed only by advancing the base or by a checker change, both of
+which are decisions this row does not take.
+
+## 6. What the other 27 commits owed, which was nothing
+
+The audit that found this defect covered all 28. Recorded so the next person
+does not repeat it.
+
+| Group | Count | Verdict |
+|---|---|---|
+| `MODEL-MM-indextts2` (#634) | 19 | **Already documented.** Intermediate waves — CAMPPlus, w2v-bert, EnhancedCodec, S2Mel DiT primitives. The campaign documented itself at `f374ab8ed` once the pipeline rendered end to end, and every flagged commit is an ancestor of it. `docs/FEATURES.md:200` records the partial state honestly, defects included |
+| `ENG-EXPERT-STREAM` (#913, #916, #918) | 3 | **Must NOT be documented.** Deliberately unwired staged slices; #918's own body says "Nothing wired to a loader; the engine is unchanged." A `docs/FEATURES.md` row would claim a capability no user can reach |
+| `ENG-WEIGHT-OFFLOAD` (#843, #877, #879, #884) | 4 | **Already documented** by `62406c30e`, which postdates them |
+| `ENG-WEIGHT-OFFLOAD` (#887) | 1 | **This row.** The only one that postdates its campaign's docs commit, and the only one where the docs are wrong rather than silent |
+| `LTX25-PROMPT-ADALN` (`fba312c67`) | 1 | **Nothing owed.** A record/measurement commit — specs, goldens, a measurement script, tests. Ships no user-facing surface |
+
+The pattern worth keeping: a staged campaign that documents itself once, at the
+point the capability becomes reachable, is the *correct* shape. The gate flags
+every wave that is not that commit, so a large flagged count is expected and is
+not by itself evidence of stale documentation. Only a commit that changes
+user-visible behaviour **after** its campaign's docs landed can make a document
+wrong, and exactly one of the 28 did.
+
+## 7. Now
+
+`main` is red on `documentation-checkpoint` for 28 commits and stays red after
+this row. What changes is that the two documents stop describing a behaviour the
+engine has not had since `4a183b731`.
+
+## Owed
+
+Nothing by this row. The wiring that would make an enabled offload succeed is
+[#797](https://github.com/mudler/vllm.cpp/issues/797) under
+`ENG-WEIGHT-OFFLOAD`; when a loader is wired, the *What works today* table row
+and the refusal subsection added here both have to move with it.
diff --git a/docs/USAGE.md b/docs/USAGE.md
@@ -1470,7 +1470,7 @@ a stop token early.
 | `--tool-call-parser <name>` | `hermes` | Tool-call dialect (42 names over 38 families). `auto` detects from the chat template, `none` disables. For `gemma4`, OpenAI chat uses the text-seam parser (wrapped `<\|tool_call>` **or** bare `call:NAME{ARGS}`) so free-form / detokenized tool bodies still become `tool_calls`. **`inkling` needs `"skip_special_tokens": false` on the request today** — its whole grammar is special tokens and we have no `adjust_request` seam to force the flag off for you, so at the `true` default the detokenizer strips the markers before the parser runs ([#695](https://github.com/mudler/vllm.cpp/issues/695)). `--reasoning-parser inkling` is not registered at all ([#703](https://github.com/mudler/vllm.cpp/issues/703)) |
 | `--reasoning-parser <name>` | `none` | Reasoning parser (`think_auto`, `deepseek_r1`, `deepseek_v3`, `holo2`, `mistral`, `minimax_m2`, `minimax_m2_append_think`, `step3`, `olmo3`, `muse_glimmer`, `qwen3`, `mimo`). `auto` detects, `none` disables. `qwen3` and its `mimo` alias are the engine-backed adapter (one upstream class, two registry names): thinking is ON, so a marker-less stream is reasoning and a `<tool_call>` ends reasoning with no `</think>`. `auto` never selects it — a generic `<think>` template resolves to `think_auto`, which is the right default for hybrid-thinking models that may answer with no think block at all |
 | `--kv-transfer-config '<json>'` | (unset) | External KV connector, same JSON as vLLM's flag. See [docs/KV-OFFLOAD.md](KV-OFFLOAD.md) |
-| `--offload-config '<json>'` | (unset) | Weight offload, the same JSON vLLM's `OffloadConfig` takes (distinct from `--kv-transfer-config`, which offloads KV blocks). Parsed and validated at startup, so a malformed document, an unknown backend or a validator violation is refused before any model I/O; a backend/field mismatch is a warning, as upstream. **Accepted and inert today: no weight moves yet**, and on unified memory such as GB10 it cannot help at all because host and device share one pool. See [docs/WEIGHT-OFFLOAD.md](WEIGHT-OFFLOAD.md) |
+| `--offload-config '<json>'` | (unset) | Weight offload, the same JSON vLLM's `OffloadConfig` takes (distinct from `--kv-transfer-config`, which offloads KV blocks). Parsed and validated at startup, so a malformed document, an unknown backend or a validator violation is refused before any model I/O; a backend/field mismatch is a warning, as upstream. **Enabling it fails startup on every model today**: no loader consults the offloader, so the engine refuses the configuration by architecture name rather than accept a budget that frees nothing. A config that leaves offloading disabled still parses and reports normally. On unified memory such as GB10 offload cannot help at all, because host and device share one pool. See [docs/WEIGHT-OFFLOAD.md](WEIGHT-OFFLOAD.md) |
 | `--speculative-config '<json>'` | (unset) | Speculative decoding (`mtp`, `dflash`, `ngram`), same JSON as vLLM's flag. `dspark` speculates on the Qwen3.6 gate models (native + Speculators drafts), token-identically to speculative-off, but is not gated on speed: the cross-engine ratio is UNSETTLED, with a matched-and-warm paired measurement of 0.834x against the pinned oracle and the earlier 0.957x-0.989x figures taken against a single COLD oracle invocation on a machine that has since been reimaged. A GGUF target, or a target with no aux multi-tap, is refused by name (`SPEC-DSPARK`). Its sequential Markov sampling runs on device by default; `VT_DSPARK_DEVICE_SAMPLE=0` restores the host loop (token-identical, cost only). The speculative verify runs from a captured CUDA graph, worth +12.2%/+3.5% on the 35B cells; `VT_SPEC_DECODE_GRAPH=0` restores the eager verify (also token-identical). See [docs/SPECULATIVE-DECODING.md](SPECULATIVE-DECODING.md) |
 | `--language-model-only` / `--no-language-model-only` | off | Disable all multimodal input by setting **every** modality limit to 0, mirroring vLLM's flag of the same name. It is not a "skip the encoder" switch: the server then **refuses** a multimodal request with ``400 At most 0 image(s) may be provided in one prompt. Set `--limit-mm-per-prompt` to increase this limit.`` It does **not** free VRAM yet — nothing gates tower construction on it ([#607](https://github.com/mudler/vllm.cpp/issues/607) wave L3) |
 | `--limit-mm-per-prompt '<json>'` | (unset ⇒ 999 per modality) | Maximum multimodal input items per prompt, per modality, as the same JSON object vLLM's flag takes: `'{"image": 2, "video": 0}'`, or with profiling options `'{"video": {"count": 1, "num_frames": 32}}'` (the options are validated and ignored — they size dummy inputs for memory profiling, which this engine does not do). A limit can only **lower** what the model/seam supports, never raise it. Malformed JSON, a negative count, or an unknown option on `image` / `video` / `audio` is refused at startup rather than defaulted. An unknown option on any other modality name is dropped rather than refused, mirroring upstream, whose fallback `BaseDummyOptions` is the one such dataclass without `extra="forbid"`. Upstream's dotted spelling (`--limit-mm-per-prompt.image 2`) is not accepted here, as for `--kv-transfer-config` and `--speculative-config` |