From 98f1fd496b8e1ebe103918ee07200234223c45e0 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 15 Aug 2026 20:53:14 +0000 Subject: [PATCH] docs(FIX-OFFLOAD-DOCS-957): weight offload is refused at startup, not accepted and inert MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes #957. No code moves. `4a183b731` (#887) turned a configured weight offload from ACCEPTED-AND-INERT into a hard startup refusal, and neither public document followed. Both still described the behaviour that commit replaced, so a reader who followed them would add `--offload-config` expecting a no-op and get a server that does not start. ## What the engine actually does `RefuseUnsupportedWeightOffload` (`weight_offloader.cpp:72-83`) throws when offloading is enabled and the resolved architecture does not declare support. It is called unconditionally from the load path (`model_loader.cpp:1410-1414`), right after `ModelRegistry::Resolve` and BEFORE any weight I/O. `ModelFactory::supports_weight_offload` defaults to false and NO model in the tree sets it, so EVERY architecture is refused today. The tree asserts this itself: `test_weight_offloader.cpp:376-379` counts the declaring models and carries the message "a model now declares supports_weight_offload; update this". Negative control on the grep: a misspelled needle returns 0, so the empty set is not a wrong search term. A second refusal from that commit was also undocumented — `VerifyWeightOffloadWasConsulted` throws after load when a model declares support and then consults the offloader about zero weights, reported as a defect in that loader rather than a configuration error. The old refusal list covered config validation only. ## Checked, not assumed The example error names `Qwen3MoeForCausalLM`, which is a real registered architecture (`qwen3_moe_registry.cpp:181`). An invented name in an error example is indistinguishable from a real one to a reader. The "only when enabled" claim is `weight_offloader.cpp:75`. The quoted message is transcribed from the throw. `docs/STATUS.md` needed no change: it already lists weight offload under what is owed and never claimed the inert behaviour. ## THIS DOES NOT MAKE documentation-checkpoint GREEN `check-doc-checkpoint.py` evaluates each commit against its OWN parent, so no later commit can satisfy an earlier one, and there is no waiver mechanism (`:256` says exempting named paths was deliberately avoided). Measured: `docs/FEATURES.md`/`docs/USAGE.md` changed in 66 commits since `73d217db`, which still reports ERROR. The 28-commit backlog is closed only by advancing the base or changing the checker, and this row takes neither decision. ## The other 27 owed nothing Audited all 28. The 19 `MODEL-MM-indextts2` commits are intermediate waves whose campaign documented itself at `f374ab8ed` once the pipeline rendered end to end, and every one is an ancestor of it. The 3 `ENG-EXPERT-STREAM` commits are deliberately unwired staged slices that `docs/FEATURES.md` must NOT claim — #918's own body says "Nothing wired to a loader". Four earlier `ENG-WEIGHT-OFFLOAD` commits predate `62406c30e`, which documented them. `fba312c67` is a record/measurement commit shipping no user surface. A staged campaign that documents itself once, when the capability becomes reachable, is the CORRECT shape. The gate flags every wave that is not that commit, so a large flagged count is expected and is not by itself evidence of stale docs. Only a commit landing user-visible behaviour AFTER its campaign's docs can make a document wrong. Exactly one of the 28 did. Spec: `.agents/specs/offload-docs-refusal.md`. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/issue-index.md | 1 + .agents/specs/offload-docs-refusal.md | 151 ++++++++++++++++++++++++++ docs/USAGE.md | 2 +- docs/WEIGHT-OFFLOAD.md | 43 ++++++-- 4 files changed, 189 insertions(+), 8 deletions(-) create mode 100644 .agents/specs/offload-docs-refusal.md diff --git a/.agents/issue-index.md b/.agents/issue-index.md index aad878c4b..3f66482e6 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -251,3 +251,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#874](https://github.com/mudler/vllm.cpp/issues/874) | — | `windows-msvc-cpu`/`windows-msvc-vulkan` still start on a CLOSED pull request: `check-release-workflow.py::validate_pr_ci` compares their whole job mapping for equality, so neither an `if:` clause nor a `needs:` guard can be added. Listed under `## Owed` in [`ci-concurrency.md`](specs/ci-concurrency.md) | bug | | [#949](https://github.com/mudler/vllm.cpp/issues/949) | — | Nothing in the tree refuses a borrowed `vt::Tensor` that outlives the object owning its storage, and the ONLY instrument that catches one is `sanitize-cpu`, which is `continue-on-error` — that is how [#904](https://github.com/mudler/vllm.cpp/issues/904) landed. Measured in the #936 review rather than argued: with the #904 fix reverted, a plain Release build with no sanitizer runs the case 18/18 passed, 546 assertions, `rc=0`, because `dtype` lives in the `vt::Tensor` struct and not in the freed buffer, so no ordinary gate can see the dangling read. Three remedies are open and none is foregone: promote the lane once it has a `main` baseline, add a test that fails without a sanitizer, or reject the pattern statically — a prototype detector for a member access chained onto a call returning an owning type by value swept 1777 files with no hit but the defect. Anchors: the owning deleter `src/vllm/model_executor/models/ltx2_device.cpp:1088 @ 800dd082f`, the read `src/vt/cpu/cpu_layernorm.cpp:33 @ 800dd082f`. Listed under `## Owed` in [`ltx2-device-staged-view-uaf.md`](specs/ltx2-device-staged-view-uaf.md) | bug | | [#933](https://github.com/mudler/vllm.cpp/issues/933) | `ENG-EXPERT-STREAM` | Measure gateability of the `llama-cpp-unsloth` oracle by BUILDING it and RUNNING `Qwen3.8-2.4T-A95B UD-Q1_0` on it. The oracle is pinned at `36fe8e1cc` (branch `iq1-narrow`) and records `gateable = no`, because the IQ1_XXXS port is grounded in the fork's SOURCE, read and cited, which is weaker than a running comparison. It is the only place ggml type 66 is defined: the vllm.cpp pin `237ad9b96` ends at `Q1_0 = 41` and `ggml-org` master `ad1de39e0` at `Q2_0 = 42`, while type 66 carries 96.92 % of that checkpoint's parameters. Running it needs the full 370 GiB checkpoint and, per Unsloth's documentation, at least 450 GB of RAM. Until then the ported arm has no running oracle, which is what `gateable = no` makes visible | task | +| [#957](https://github.com/mudler/vllm.cpp/issues/957) | `FIX-OFFLOAD-DOCS-957` | `4a183b731` (#887) turned a configured weight offload from ACCEPTED-AND-INERT into a hard startup refusal, and neither public document followed. `docs/WEIGHT-OFFLOAD.md` still said "a budget you set is accepted, reported, and does not free memory" and `docs/USAGE.md:1473` still said "Accepted and inert today", while `RefuseUnsupportedWeightOffload` (`src/vllm/model_executor/weight_offloader.cpp:72-83` @ 2daa3287f) throws from the load path (`src/vllm/entrypoints/model_loader.cpp:1410-1414` @ 2daa3287f) before any weight I/O. `ModelFactory::supports_weight_offload` defaults false and NO model sets it, so every architecture is refused; `tests/vllm/model_executor/test_weight_offloader.cpp:376-379` @ 2daa3287f asserts that count itself. Found auditing the 28 commits `documentation-checkpoint` flags: 27 needed nothing | bug | diff --git a/.agents/specs/offload-docs-refusal.md b/.agents/specs/offload-docs-refusal.md new file mode 100644 index 000000000..8696c5eed --- /dev/null +++ b/.agents/specs/offload-docs-refusal.md @@ -0,0 +1,151 @@ +# The docs describe a weight offload that is accepted and inert; the engine refuses it + +Row: `FIX-OFFLOAD-DOCS-957` +Issue: [#957](https://github.com/mudler/vllm.cpp/issues/957) +Introduced by: `4a183b731` (#887, `ENG-WEIGHT-OFFLOAD` W2c) +Baseline: `origin/main` @ `2daa3287f` + +## 1. Scope + +Correct `docs/WEIGHT-OFFLOAD.md` and `docs/USAGE.md` so they describe what the +engine does today: refuse an enabled weight offload at startup, on every +architecture, naming the architecture. + +**Out of scope:** wiring a loader (that is `ENG-WEIGHT-OFFLOAD` W2c, issue +[#797](https://github.com/mudler/vllm.cpp/issues/797)), and changing any +refusal. This row moves no code. It also does not attempt to make +`documentation-checkpoint` green — §5 explains why nothing can. + +## 2. What the documents said, and what the engine does + +Both documents describe the behaviour `4a183b731` replaced. + +`docs/WEIGHT-OFFLOAD.md`, the callout and the summary: + +> A budget you set is accepted, reported, and does not free memory. + +> So the honest summary is: you can configure weight offload, the engine tells +> you what it resolved, and your device memory does not change. The engine +> prints one line at startup saying so [...] + +`docs/USAGE.md:1473`: + +> **Accepted and inert today: no weight moves yet** + +What actually happens, at `2daa3287f`: + +`RefuseUnsupportedWeightOffload` +(`src/vllm/model_executor/weight_offloader.cpp:72-83`) throws +`std::invalid_argument` when offloading is enabled and the resolved +architecture's factory does not set `supports_weight_offload`. It is called +unconditionally from the load path +(`src/vllm/entrypoints/model_loader.cpp:1410-1414`), immediately after +`ModelRegistry::Resolve(config)` and **before any weight I/O** — so nothing is +read from disk before the operator is told. + +`ModelFactory::supports_weight_offload` defaults to `false` +(`include/vllm/model_executor/models/model_registry.h:371`), and **no model in +the tree sets it to `true`**. Measured by grepping `src/` and `include/` for the +identifier: every hit is the declaration, the two guard parameters, or the +`model_loader.cpp` read. A deliberately misspelled needle returns 0, so the +search term is not the reason the set is empty. The tree asserts the same fact +independently — `tests/vllm/model_executor/test_weight_offloader.cpp:376-379` +counts the declaring models and carries the message "a model now declares +supports_weight_offload; update this". + +So the practical statement is the one neither document made: **enabling weight +offload fails startup, for every model.** A reader following the old text would +add `--offload-config` expecting a no-op and get a server that does not start. + +A second refusal from the same commit was also undocumented. +`VerifyWeightOffloadWasConsulted` (`weight_offloader.cpp:86-97`) throws after +load when a model declares support and then asks the offloader about zero +weights, and reports it as a defect in that loader rather than a configuration +error. The document's refusal list covered only config validation — malformed +document, unknown backend, wrong type, negative budget, `offload_num_in_group` +above `offload_group_size`, `offload_prefetch_step` below 1. + +## 3. The change + +`docs/WEIGHT-OFFLOAD.md`: + +- the callout leads with the startup failure instead of "accepted, reported, and + does not free memory"; +- the *What works today* table gains a row for the model gate, so the table + answers "can I run with this on?" rather than only "what is built?"; +- the summary says you can write and validate a configuration and cannot yet run + with one enabled; +- a new subsection under *What the engine refuses* carries the real message text + and the post-load consistency check. + +`docs/USAGE.md`: the `--offload-config` row says enabling it fails startup on +every model, and that a config which leaves offloading disabled still parses and +reports. + +Both keep the existing point that on unified memory such as GB10 offload cannot +help regardless, because host and device share one pool. + +`docs/STATUS.md` needed no change: it already lists weight offload under what is +owed and never claimed the inert behaviour. + +## 4. Evidence + +No code moves, so the gate is the accuracy of the text against the tree. + +| Claim in the new text | How it was checked at `2daa3287f` | +|---|---| +| the refusal is on the load path, before weight I/O | read `model_loader.cpp:1403-1415`; the call sits between `ModelRegistry::Resolve` and `LoadShards` | +| it fires only when offloading is enabled | `weight_offloader.cpp:75` — `if (!config.is_offloading_enabled()) return;` | +| the quoted message is the real one | transcribed from the `throw` at `weight_offloader.cpp:77-83` | +| `Qwen3MoeForCausalLM` is a real architecture string | `REGISTER_VLLM_MODEL(qwen3_moe, "Qwen3MoeForCausalLM", ...)` at `qwen3_moe_registry.cpp:181` — an invented name in an error example would be indistinguishable from a real one to a reader | +| no model declares support | grep over `src/` and `include/`; all hits are declaration, parameters, or the loader read. Negative control: a misspelled needle returns 0 | +| the post-load check exists and is worded as a loader defect | `weight_offloader.cpp:86-97` | + +## 5. This does not make `documentation-checkpoint` green, and nothing can + +`scripts/check-doc-checkpoint.py` walks the range one commit at a time and +evaluates each commit's **own** paths against its **own** parent (`main()`, the +`--base`/`--head` branch, via `commit_paths(commit)`). No later commit can +satisfy an earlier one, and there is no waiver or exemption mechanism — the file +says at `:256` that exempting named paths was deliberately avoided. + +Measured rather than reasoned: `docs/FEATURES.md` and `docs/USAGE.md` have +changed in **66 commits** since `73d217db` landed, and +`check-doc-checkpoint.py --commit 73d217db` still reports ERROR. + +The 28 flagged commits are immutable on `main`, which is never rewritten. So the +backlog is closed only by advancing the base or by a checker change, both of +which are decisions this row does not take. + +## 6. What the other 27 commits owed, which was nothing + +The audit that found this defect covered all 28. Recorded so the next person +does not repeat it. + +| Group | Count | Verdict | +|---|---|---| +| `MODEL-MM-indextts2` (#634) | 19 | **Already documented.** Intermediate waves — CAMPPlus, w2v-bert, EnhancedCodec, S2Mel DiT primitives. The campaign documented itself at `f374ab8ed` once the pipeline rendered end to end, and every flagged commit is an ancestor of it. `docs/FEATURES.md:200` records the partial state honestly, defects included | +| `ENG-EXPERT-STREAM` (#913, #916, #918) | 3 | **Must NOT be documented.** Deliberately unwired staged slices; #918's own body says "Nothing wired to a loader; the engine is unchanged." A `docs/FEATURES.md` row would claim a capability no user can reach | +| `ENG-WEIGHT-OFFLOAD` (#843, #877, #879, #884) | 4 | **Already documented** by `62406c30e`, which postdates them | +| `ENG-WEIGHT-OFFLOAD` (#887) | 1 | **This row.** The only one that postdates its campaign's docs commit, and the only one where the docs are wrong rather than silent | +| `LTX25-PROMPT-ADALN` (`fba312c67`) | 1 | **Nothing owed.** A record/measurement commit — specs, goldens, a measurement script, tests. Ships no user-facing surface | + +The pattern worth keeping: a staged campaign that documents itself once, at the +point the capability becomes reachable, is the *correct* shape. The gate flags +every wave that is not that commit, so a large flagged count is expected and is +not by itself evidence of stale documentation. Only a commit that changes +user-visible behaviour **after** its campaign's docs landed can make a document +wrong, and exactly one of the 28 did. + +## 7. Now + +`main` is red on `documentation-checkpoint` for 28 commits and stays red after +this row. What changes is that the two documents stop describing a behaviour the +engine has not had since `4a183b731`. + +## Owed + +Nothing by this row. The wiring that would make an enabled offload succeed is +[#797](https://github.com/mudler/vllm.cpp/issues/797) under +`ENG-WEIGHT-OFFLOAD`; when a loader is wired, the *What works today* table row +and the refusal subsection added here both have to move with it. diff --git a/docs/USAGE.md b/docs/USAGE.md index 0daf7a380..97195e848 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -1470,7 +1470,7 @@ a stop token early. | `--tool-call-parser ` | `hermes` | Tool-call dialect (42 names over 38 families). `auto` detects from the chat template, `none` disables. For `gemma4`, OpenAI chat uses the text-seam parser (wrapped `<\|tool_call>` **or** bare `call:NAME{ARGS}`) so free-form / detokenized tool bodies still become `tool_calls`. **`inkling` needs `"skip_special_tokens": false` on the request today** — its whole grammar is special tokens and we have no `adjust_request` seam to force the flag off for you, so at the `true` default the detokenizer strips the markers before the parser runs ([#695](https://github.com/mudler/vllm.cpp/issues/695)). `--reasoning-parser inkling` is not registered at all ([#703](https://github.com/mudler/vllm.cpp/issues/703)) | | `--reasoning-parser ` | `none` | Reasoning parser (`think_auto`, `deepseek_r1`, `deepseek_v3`, `holo2`, `mistral`, `minimax_m2`, `minimax_m2_append_think`, `step3`, `olmo3`, `muse_glimmer`, `qwen3`, `mimo`). `auto` detects, `none` disables. `qwen3` and its `mimo` alias are the engine-backed adapter (one upstream class, two registry names): thinking is ON, so a marker-less stream is reasoning and a `` ends reasoning with no ``. `auto` never selects it — a generic `` template resolves to `think_auto`, which is the right default for hybrid-thinking models that may answer with no think block at all | | `--kv-transfer-config ''` | (unset) | External KV connector, same JSON as vLLM's flag. See [docs/KV-OFFLOAD.md](KV-OFFLOAD.md) | -| `--offload-config ''` | (unset) | Weight offload, the same JSON vLLM's `OffloadConfig` takes (distinct from `--kv-transfer-config`, which offloads KV blocks). Parsed and validated at startup, so a malformed document, an unknown backend or a validator violation is refused before any model I/O; a backend/field mismatch is a warning, as upstream. **Accepted and inert today: no weight moves yet**, and on unified memory such as GB10 it cannot help at all because host and device share one pool. See [docs/WEIGHT-OFFLOAD.md](WEIGHT-OFFLOAD.md) | +| `--offload-config ''` | (unset) | Weight offload, the same JSON vLLM's `OffloadConfig` takes (distinct from `--kv-transfer-config`, which offloads KV blocks). Parsed and validated at startup, so a malformed document, an unknown backend or a validator violation is refused before any model I/O; a backend/field mismatch is a warning, as upstream. **Enabling it fails startup on every model today**: no loader consults the offloader, so the engine refuses the configuration by architecture name rather than accept a budget that frees nothing. A config that leaves offloading disabled still parses and reports normally. On unified memory such as GB10 offload cannot help at all, because host and device share one pool. See [docs/WEIGHT-OFFLOAD.md](WEIGHT-OFFLOAD.md) | | `--speculative-config ''` | (unset) | Speculative decoding (`mtp`, `dflash`, `ngram`), same JSON as vLLM's flag. `dspark` speculates on the Qwen3.6 gate models (native + Speculators drafts), token-identically to speculative-off, but is not gated on speed: the cross-engine ratio is UNSETTLED, with a matched-and-warm paired measurement of 0.834x against the pinned oracle and the earlier 0.957x-0.989x figures taken against a single COLD oracle invocation on a machine that has since been reimaged. A GGUF target, or a target with no aux multi-tap, is refused by name (`SPEC-DSPARK`). Its sequential Markov sampling runs on device by default; `VT_DSPARK_DEVICE_SAMPLE=0` restores the host loop (token-identical, cost only). The speculative verify runs from a captured CUDA graph, worth +12.2%/+3.5% on the 35B cells; `VT_SPEC_DECODE_GRAPH=0` restores the eager verify (also token-identical). See [docs/SPECULATIVE-DECODING.md](SPECULATIVE-DECODING.md) | | `--language-model-only` / `--no-language-model-only` | off | Disable all multimodal input by setting **every** modality limit to 0, mirroring vLLM's flag of the same name. It is not a "skip the encoder" switch: the server then **refuses** a multimodal request with ``400 At most 0 image(s) may be provided in one prompt. Set `--limit-mm-per-prompt` to increase this limit.`` It does **not** free VRAM yet — nothing gates tower construction on it ([#607](https://github.com/mudler/vllm.cpp/issues/607) wave L3) | | `--limit-mm-per-prompt ''` | (unset ⇒ 999 per modality) | Maximum multimodal input items per prompt, per modality, as the same JSON object vLLM's flag takes: `'{"image": 2, "video": 0}'`, or with profiling options `'{"video": {"count": 1, "num_frames": 32}}'` (the options are validated and ignored — they size dummy inputs for memory profiling, which this engine does not do). A limit can only **lower** what the model/seam supports, never raise it. Malformed JSON, a negative count, or an unknown option on `image` / `video` / `audio` is refused at startup rather than defaulted. An unknown option on any other modality name is dropped rather than refused, mirroring upstream, whose fallback `BaseDummyOptions` is the one such dataclass without `extra="forbid"`. Upstream's dotted spelling (`--limit-mm-per-prompt.image 2`) is not accepted here, as for `--kv-transfer-config` and `--speculative-config` | diff --git a/docs/WEIGHT-OFFLOAD.md b/docs/WEIGHT-OFFLOAD.md index 4d0ca0f85..08ff134dc 100644 --- a/docs/WEIGHT-OFFLOAD.md +++ b/docs/WEIGHT-OFFLOAD.md @@ -9,9 +9,10 @@ Selection mirrors vLLM's own configuration: `--offload-config ''`, taking the same JSON object vLLM's `OffloadConfig` takes, so a config written for vLLM is accepted here. -> **Read [What works today](#what-works-today) before you enable anything.** The -> configuration surface is complete and validated, and **no weight moves yet**. -> A budget you set is accepted, reported, and does not free memory. +> **Enabling weight offload fails startup today, on every model.** No loader +> consults the offloader yet, so the engine refuses the configuration rather +> than accept a budget that would free nothing. Read +> [What works today](#what-works-today) before you enable anything. ## What works today @@ -21,12 +22,17 @@ is accepted here. | The `uva` backend object and its byte budget | Built. It answers the offload decision and counts approved bytes | | The `prefetch` backend | Not built. A config that selects it is accepted and reported as unbuilt | | A loader that asks the offloader and keeps a weight off the device | **Not wired.** This is the part that frees memory, and it does not exist yet | +| A model that accepts an enabled offload | **None.** Every architecture is refused at startup — see [What the engine refuses](#what-the-engine-refuses-and-what-it-only-warns-about) | | Pinned host copies and device views | Not built | -So the honest summary is: you can configure weight offload, the engine tells you -what it resolved, and your device memory does not change. The engine prints one -line at startup saying so, because a budget that silently frees nothing is worse -than a budget the engine refuses. +So the honest summary is: you can write and validate a weight-offload +configuration, and you cannot yet run with one enabled. A budget that silently +frees nothing is worse than a budget the engine refuses, so the engine refuses +it. + +A configuration that leaves offloading **disabled** is unaffected: the guard +only fires when `is_offloading_enabled()` is true, so parsing, validation, and +the resolved-backend report all still work for inspecting a config. Progress is tracked in [issue #797](https://github.com/mudler/vllm.cpp/issues/797). @@ -90,6 +96,29 @@ Refusals cover a malformed document, an unknown backend name, a field of the wrong type, a negative budget, `offload_num_in_group` above `offload_group_size`, and `offload_prefetch_step` below 1 while prefetch is enabled. +### The model has to claim support, and none does yet + +A valid configuration is still refused when the resolved architecture's loader +does not consult the offloader. This is the refusal you will actually hit, since +**no model declares support today**: + +```text +weight offload is configured but architecture "Qwen3MoeForCausalLM" does not +support it: its loader does not consult the weight offloader, so every weight +would stay on the device and the budget would free nothing +(ENG-WEIGHT-OFFLOAD W2c). Remove --offload-config, or wire this model's loader. +``` + +It is raised after the architecture resolves and **before any weight I/O**, so +nothing is read from disk before you are told. The support flag defaults to off, +which means a newly added model is refused until someone wires its loader — +the default is the mechanism, not an oversight. + +A model that *claims* support and then never asks the offloader about a single +weight is refused too, after load, and reported as a defect in that loader +rather than as a configuration error. Zero consulted weights is the only count +that can prove that particular lie. + A backend that disagrees with the fields you set is a **warning**, not a refusal, which mirrors vLLM. The named backend wins and the other fields are ignored: