feat(#810 A2-R): NemotronH's dense weights reach the device, and the attention block that reads them is model-local (#517)

.agents/style/commits.md

@@ -109,6 +109,18 @@ the issue, for example `Closes #123`.
 Do not use a screenshot as the only description. Images can become unavailable
 and cannot replace searchable text.
 
+**Never put a bare `---` line in a pull request body.** The repository sets
+`squash_merge_commit_message = PR_BODY`, so the body becomes the landed commit
+message, and `git interpret-trailers` treats a bare `---` as the end of that
+message. Everything below it — including the trailer block — becomes invisible
+to the parser, so `commit-protocol-tag` reports trailers the body plainly
+carries as missing. A markdown horizontal rule is the usual way this happens.
+Measured on pull request #950: two `---` rules, trailer block present and
+correct at the end, `git interpret-trailers --parse` returned nothing; deleting
+the two rules returned all three trailers. Use a heading to separate sections
+instead. A markdown table's `|---|---|` separator is not affected, because the
+line is not bare.
+
 ## Name branches
 
 For a claimed row, use the binding `row/<ROW-ID>` format. For other work, follow

CMakeLists.txt

@@ -879,6 +879,10 @@ add_library(vllm STATIC
   # config descent, the on-disk name map, the registry entry and the het-KV
   # topology. No forward: nemotron_h.cpp is W4.
   src/vllm/model_executor/models/nemotron_h.cpp
+  # A2-R (#810): the DEVICE arm — the residual stream, the 52 norms, the
+  # embedding lookup and the 6 GQA attention blocks. Mamba/MoE/lm_head stay
+  # host; see the file header for why the line falls where it does.
+  src/vllm/model_executor/models/nemotron_h_device.cpp
   src/vllm/model_executor/models/nemotron_h_registry.cpp
   src/vllm/model_executor/models/nemotron_h_weights.cpp
   src/vllm/model_executor/models/glm4_registry.cpp

docs/FEATURES.md

@@ -138,7 +138,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks.
 | `LagunaForCausalLM` | poolside/Laguna-S-2.1-NVFP4, GGUF-Q4_K, Laguna-XS | byte-exact near-tie (distributional vs vLLM) | vLLM parity+ 1.03x, default on, via the `laguna-gen` CLI; the registered engine forward VT_CHECKs non-bf16 (`ARCH-ONE-SURFACE` fold) |
 | `KimiLinearForCausalLM` | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE) | **Folded onto the shared paged runner (ROW 7 §21, #122): engine==CLI 128/128 byte-identical; vs golden 122/128 (the intrinsic near-tie profile); FA2 paged MLA default-ON; SACRED post-fold green** | Served via `vllm_engine_load` + `vllm_complete_tokens` (ABI v13); server 19.0 tok/s wall vs vLLM ~21 (~0.90×), speed residual open |
 | `KimiK3ForConditionalGeneration` | Kimi-K3 (2.8T MoE) | scaffold: registry+config+enumeration gated, forward refuses | HW-infeasible (~1.56 TB); no run |
-| `NemotronHForCausalLM` | Nemotron-3.5-Lightning-30B-A3B-NVFP4 (`nvidia` @`29f2d174`) | config+enumeration+KV-shape gated; hybrid Mamba2/GQA/relu2-MoE forward COMPUTES. Loader materializes 18487/18487 as SHIPPED (5935 NVFP4 g16, 46 FP8 W8A8, bf16); 270 MTP owed to W5 (#517) | The paged runner ALLOCATES it from the model's OWN KV spec, so engine construction no longer refuses under Qwen3.5's name; the step refuses by name until A2 (#810). Host forward: 17.7 GiB RSS, 3/3 first tokens |
+| `NemotronHForCausalLM` | Nemotron-3.5-Lightning-30B-A3B-NVFP4 (`nvidia` @`29f2d174`) | config+enumeration+KV-shape gated; hybrid Mamba2/GQA/relu2-MoE forward COMPUTES. Loader materializes 18487/18487 as SHIPPED (5935 NVFP4 g16, 46 FP8 W8A8, bf16); 270 MTP owed to W5 (#517) | Engine construction allocates from the model's OWN KV spec (#810 A1); the step still refuses by name. A2-R adds a PARTIAL device arm (embeddings, 52 norms, 6 GQA blocks); Mamba2/MoE/lm_head stay host, no speed claim |
 | `MuseGlimmerForCausalLM` | real tensors, **bf16 depth 4/52 only**: 5 prefill argmax positions match a torch transcription of vllm#51655 and HF. GGUF full depth generates coherently (#347, #359) but is **NOT token-exact** | text forward + loader vs an fp32 reference, per-mechanism property tests, scaffold 11/11, GGUF gate 17/17. An ABSENT config key now takes the architecture's constant (#412): GGUF post-norms ran at 1e-5, not 1e-8 | no vLLM denominator (pin cannot load it); SECONDARY llama.cpp, same GGUF, GB10 CPU: prefill tie **0.997x**, decode 0.232x, RSS 1.92x (#333) |
 | `MuseGlimmerForConditionalGeneration` | vision: **no reference run of any kind**; enumeration gated vs the released 30B index (1436/1436). Image/video need bf16 safetensors: `mmproj-kquant.gguf` is refused by name | perception encoder loaded and wired, so an image or video prompt runs; `perception_emb_norm` now armed by default (#405). Reachability plus placeholder scatter only, no image or video correctness | not measurable; anchored to open vllm#51655 |
 | `LlamaModel` | landed tiny synthetic embedding fixture (engine path == direct pooler path, identical vectors; f64 LAST+normalize reference); real checkpoint (e5-mistral class) is a NAMED residual | pooling/embed only, text paths refuse by task; `vllm_embed` + `/v1/embeddings` | n/a (CPU correctness-grade embeddings) |

docs/USAGE.md

@@ -456,7 +456,7 @@ tokens quietly.
 | Architecture | Why it refuses |
 |---|---|
 | `KimiK3ForConditionalGeneration` | Needs ~1.56 TB (MXFP4); no host here can run it |
-| `NemotronHForCausalLM` | The hybrid forward is ported (#517 W4) and the weight loader materializes the real checkpoint, but that forward is a HOST reference: it recomputes K/V over the whole sequence every step, carries no recurrent state between steps and treats a batch as one causal sequence. Engine construction now SUCCEEDS — the KV allocation reads the model's own recurrent spec (#810) — and the first step then refuses by name, naming the paged/batched decode path as the missing piece rather than returning plausible wrong tokens. Safetensors resolve and parse; a GGUF file is refused by name, since no GGUF arm exists for it |
+| `NemotronHForCausalLM` | The hybrid forward is ported (#517 W4) and the weight loader materializes the real checkpoint, but that forward is a HOST reference: it recomputes K/V over the whole sequence every step, carries no recurrent state between steps and treats a batch as one causal sequence. Engine construction now SUCCEEDS — the KV allocation reads the model's own recurrent spec (#810) — and the first step then refuses by name, naming the paged/batched decode path as the missing piece rather than returning plausible wrong tokens. **That refusal is UNCHANGED by A2-R (#810)**: A2-R adds a partial device arm (embedding lookup, the 52 layer norms + `norm_f`, and the 6 GQA attention blocks; Mamba2, MoE and `lm_head` stay on the host), but it is non-paged and single-request, so it creates none of the capability the refusal guards and is not reachable through `include/vllm.h`. It is exercised only by `test_nemotron_h_forward`, and it records no throughput number. Safetensors resolve and parse; a GGUF file is refused by name, since no GGUF arm exists for it |
 
 This is a deliberate state, not a bug: registering the architecture is what lets
 the config parse and weight-name mapping be tested before the forward exists.

src/vllm/model_executor/models/nemotron_h.cpp

@@ -177,6 +177,26 @@ Tensor F32V(std::vector<float>& v, vt::Device dev, std::vector<int64_t> shape) {
   return t;
 }
 
+// A2-R (#810): the same three checks for the weights that moved to the shared
+// `OwnedTensor` residency type (embeddings, the 53 norms, attention q/k/v/o).
+// An overload rather than a template so the refusal messages stay byte-identical
+// to the NemotronHOwned arm below — diffing the two should show one difference,
+// the shape accessor.
+void RequireWeight(const OwnedTensor& w, const char* what, DType want,
+                   std::vector<int64_t> shape) {
+  VT_CHECK(!w.Empty(), std::string("NemotronHForCausalLM forward: weight '") + what +
+                           "' is not materialized (the safetensors/quantized "
+                           "weight load is owed; see "
+                           ".agents/specs/nemotron-h-model.md §5b)");
+  VT_CHECK(w.dtype == want,
+           std::string("NemotronHForCausalLM forward: weight '") + what +
+               "' has the wrong dtype for this arm");
+  VT_CHECK(w.rank == static_cast<int>(shape.size()) &&
+               std::equal(shape.begin(), shape.end(), w.shape),
+           std::string("NemotronHForCausalLM forward: weight '") + what +
+               "' has the wrong shape");
+}
+
 void RequireWeight(const NemotronHOwned& w, const char* what, DType want,
                    std::vector<int64_t> shape) {
   VT_CHECK(!w.Empty(), std::string("NemotronHForCausalLM forward: weight '") + what +
@@ -279,6 +299,32 @@ Buf Linear(Queue& q, const Buf& a, const NemotronHOwned& w, int64_t M, int64_t K
   return out;
 }
 
+// A2-R: the OwnedTensor arm. There is NO DenseFor here and that is the point —
+// an OwnedTensor weight is dense by construction, so this arm has no dequant
+// branch to take and cannot silently widen anything. `View()` yields a host/CPU
+// tensor, which is what this HOST reference forward's queue is.
+Buf Linear(Queue& q, const Buf& a, const OwnedTensor& w, int64_t M, int64_t K,
+           int64_t N, const char* what) {
+  RequireWeight(w, what, a.dtype, {N, K});
+  // `nk` IS CONSUMED HERE, not merely recorded. `vt::MatmulBT` reads `b` as
+  // [N=out, K=in] — the raw torch-Linear orientation `nk = true` names
+  // (qwen3_5_weights.h:57-61). A weight the loader recorded as [K, N] is a
+  // TRANSPOSED GEMM operand: same byte count, same shape, a plausible wrong
+  // answer. Refused by name, as qwen3_5.cpp:3353 / :3611 / :7638 refuse the
+  // same thing for their own MatmulBT operands. Before this check `nk` had no
+  // reader on this path at all, so flipping it in the loader changed nothing
+  // any gate could see.
+  VT_CHECK(w.nk, std::string("NemotronHForCausalLM forward: weight '") + what +
+                     "' is not in the [out, in] torch-Linear orientation "
+                     "vt::MatmulBT consumes");
+  Buf out(a.dtype, {M, N});
+  Tensor at = a.t(q.device, {M, K});
+  Tensor wt = w.View();
+  Tensor ot = out.t(q.device);
+  vt::MatmulBT(q, ot, at, wt);
+  return out;
+}
+
 // Copy a column range [c0, c0+width) out of `src [rows, src_cols]` into a fresh
 // CONTIGUOUS buffer. `vt::Mamba2ChunkScan` validates every operand contiguous
 // (ops.cpp CheckMamba2Operand), so the fused zxbcdt / xBC splits cannot be
@@ -851,7 +897,7 @@ std::vector<float> NemotronHForward(const NemotronHHostWeights& host,
       VT_CHECK(id >= 0 && id < V, "NemotronHForCausalLM forward: token id out of range");
     }
     Tensor ot = residual.t(dev);
-    Tensor tab = host.embeddings.View(dev);
+    Tensor tab = host.embeddings.View();
     Tensor it = I32(ids, dev, {T});
     vt::Embedding(queue, ot, tab, it);
   }
@@ -880,7 +926,7 @@ std::vector<float> NemotronHForward(const NemotronHHostWeights& host,
     Buf normed(adt, {T, H});
     {
       Tensor ot = normed.t(dev);
-      Tensor wt = lw.norm.View(dev);
+      Tensor wt = lw.norm.View();
       if (l == 0) {
         // `residual is None` (nemotron_h.py:627-631): the embedding IS the
         // residual and the norm is UN-fused, so there is no add to fuse here.
@@ -933,7 +979,7 @@ std::vector<float> NemotronHForward(const NemotronHHostWeights& host,
     Tensor ot = final_normed.t(dev);
     Tensor xt = carry.t(dev);
     Tensor rt = residual.t(dev);
-    Tensor wt = host.norm_f.View(dev);
+    Tensor wt = host.norm_f.View();
     if (FusedChainAdoptEnabled()) {
       vt::FusedChain(queue, ot, xt, wt, &rt, vt::kFusedAddRmsNormStd,
                      static_cast<float>(params.layer_norm_epsilon));
@@ -963,7 +1009,33 @@ std::vector<float> NemotronHForward(const NemotronHHostWeights& host,
                     static_cast<size_t>(want[static_cast<size_t>(r)] * H) * esz,
                 static_cast<size_t>(H) * esz);
   }
-  const Buf logits = Linear(queue, gathered, host.lm_head, R, H, V, "lm_head.weight");
+  // A2-R: through the SHARED projection, so the host reference and the device
+  // arm cannot drift apart in their output layer. `UnpackF32` here and
+  // `PackF32` inside the helper are exact inverses for both admitted dtypes
+  // (bf16->f32->bf16 is lossless; f32 is the identity), so this is
+  // bit-identical to the direct `Linear` call it replaces.
+  return NemotronHHostLmHead(host, params, UnpackF32(gathered), R, queue);
+}
+
+std::vector<float> NemotronHHostLmHead(const NemotronHHostWeights& host,
+                                       const NemotronHParams& params,
+                                       const std::vector<float>& gathered_normed,
+                                       int64_t num_rows, vt::Queue& queue) {
+  CheckActDType(host.act_dtype);
+  const DType adt = host.act_dtype;
+  const int64_t H = params.hidden_size;
+  const int64_t V = params.vocab_size;
+  VT_CHECK(num_rows > 0, "NemotronH lm_head: no rows requested");
+  VT_CHECK(static_cast<int64_t>(gathered_normed.size()) == num_rows * H,
+           "NemotronH lm_head: gathered row count does not match hidden_size");
+  VT_CHECK(queue.device.type == vt::DeviceType::kCPU,
+           "NemotronH lm_head: this projection is the HOST arm and requires a "
+           "CPU queue — `lm_head` is NVFP4 W4A16 g16 on the released checkpoint "
+           "and the device NVFP4 arm is not ported (#810 A2-Q)");
+  RequireWeight(host.lm_head, "lm_head.weight", adt, {V, H});
+  const Buf gathered = PackF32(gathered_normed, adt, {num_rows, H});
+  const Buf logits =
+      Linear(queue, gathered, host.lm_head, num_rows, H, V, "lm_head.weight");
   return UnpackF32(logits);
 }