feat(#810 A2-R): NemotronH's dense weights reach the device, and the attention block that reads them is model-local (#517)

[localai-bot]

Implements A2-R of .agents/specs/nemotron-h-abi-e2e.md, issue #810, parent row #517.

A2 was scoped as one unit and came back as a campaign. This is the first landable piece: the weights the device can actually execute move onto the tree's shared residency seam, and the 6 GQA attention layers compute there.

⚠ The unit names, because the obvious ones collide

The spec's §1 "permitted split" (lines 236-241) already defines an A2a and an A2b, and they are not this campaign's units. The spec's A2a is the paged unit — the one that narrows the G-SAFE interlock. An earlier revision of this PR also called itself A2a, so a reader grepping A2a landed here and expected the interlock to have moved. It has not. A2b is taken twice over: the spec's batching unit, and Laguna's unrelated "Brick A2b" (src/vt/cuda/cuda_laguna.cu:368).

name	what it is	interlock
A2-R	THIS unit. Dense weights on the shared OwnedTensor seam; embeddings, 52 norms + norm_f and the 6 GQA blocks on the device	untouched
A2-Q	the NVFP4 / FP8 device arms (nemotron_h.cpp:1023)	untouched
A2-P	the spec's A2a — single-request paged decode	narrows it to num_reqs > 1
A2-B	the spec's A2b — batching	drops the num_reqs clause

The rename is in the tree, not only here: the three TEST_CASE names, all 36 former Greek references across the model sources, the CMakeLists.txt source entry, both doc rows, and this PR's title. The mapping is recorded at tests/vllm/models/test_nemotron_h_forward.cpp §8 so it is greppable.

src/vllm/model_executor/models/nemotron_h_registry.cpp is the one deliberate exception. It still says A2b, in the spec's own sense, and the file is byte-identical to main: it carries the G-SAFE interlock, and an empty diff on it is evidence a reviewer reads directly. Touching it to fix a comment would cost that.

What runs where, and why the line falls there

DEVICE	the embedding lookup, all 52 layer norms + norm_f, the 6 GQA attention blocks. The residual stream is device-resident for the whole forward.
HOST	the 23 Mamba2 blocks, the 23 MoE blocks, lm_head.

The line is drawn by the memory format the checkpoint ships, not by taste. The released checkpoint is MIXED_PRECISION and only 216 of its tensors are plain bf16 (gated: test_nemotron_h_loader.cpp:254). Everything put on the device comes from that population; everything left on the host does not:

• The 23 Mamba2 blocks are entered through mixer.in_proj, FP8 W8A8. The block is not splittable — in_proj produces the fused zxbcdt that both the conv and the scan consume — so it moves as a unit, after the shared FP8 W8A8 linear seam is extracted out of qwen3_5.cpp (ResidentFp8:1456, MatmulFp8CutlassD:1495). That is The FP8 W8A8 linear path is not a shared seam: residency and GEMM entry points live inside qwen3_5.cpp, so a second model cannot reach them without hand-rolling a parallel path #940, deliberately not done here: its own gate is Qwen3.5 byte-identity, which does not belong in a NemotronH row.
• The 5934 expert projections and lm_head are NVFP4 W4A16 g16: 30.19e9 parameters, 15.8 GiB packed, 56.2 GiB dequantized to bf16. There is no device NVFP4→bf16 dequant kernel in vt at all (the only standalone device dequant is vt::DequantFp8ChannelBf16, fused_ops.h:31, ROCm-only and per-channel FP8), so a "bf16 everywhere" device forward means a host dequant plus a 56.2 GiB upload. Both gate hosts are unified-memory, so that is a reboot, not an OOM. nemotron_h_loader.h:36-46 rejected the same design for the load.

What this does NOT prove

46 of 52 layers still compute on the host. lm_head still computes on the host. Nothing here is paged, nothing batches, and no throughput number is recorded or implied — this arm is slower than the host reference, because each host layer costs one download plus one upload. That bounce is scaffold, not architecture; every later unit deletes one pair of it.

Nothing production-facing reaches the new code. NemotronHDeviceForward and NemotronHAttnBlockHostIO are called only from test_nemotron_h_forward; ForwardNemotronHForCausalLM still routes to the host NemotronHForward. That is disclosed rather than hidden, and it is A2-P that must wire the device arm through ModelRegistry::Forward or the arm stays dead. Owed, named, and tracked on #810.

G-SAFE is fully intact

All three clauses at nemotron_h_registry.cpp:161-170 — attn_kv.empty(), gdn_state.empty(), num_reqs <= 1 — are untouched. The branch does not modify that file at all. This arm is non-paged and single-request, so it creates none of the capability the interlock guards and therefore does not narrow it.

Not dense_attn::AttnBlock (#941)

The spec's §2 names it as this model's seam. That is wrong, and routing there would reintroduce the exact defect #810 removed from the runner — a shared function reading HF-config fields this architecture does not ship. Three measured reasons:

1. It takes Qwen3DenseAttnWeights (dense_attn_block.h:335), not this model's separate q/k/v/o.
2. It reads cfg.rms_norm_eps. This config ships layer_norm_epsilon and norm_eps and no rms_norm_eps, which hf_config.cpp:551 defaults to 0.0 — a silent eps=0 normalization.
3. Its default path calls vt::RopeNeox unconditionally (:496) against kNemotronHAttentionHasNoRope.

NemotronHAttnBlock follows the tree's model-local idiom (granite.cpp:84, gemma.cpp:42, glm4.cpp:80, none allowlisted) and documents its six deltas.

Not a d_dev on NemotronHOwned

That was the smaller diff and it is the parallel path AGENTS.md forbids. Dense weights move to the shared OwnedTensor and upload through dense_attn::ResidentWeight. NemotronHOwned survives for the quantized weights, precisely because OwnedTensor carries no form, no group scale and no input_scale — converting a quantized weight to it would discard the memory format the checkpoint ships.

Review repairs (findings on f77dbac3f)

Four of the seven findings were one defect wearing different clothes: an assertion that cannot fail reads exactly like one that passes.

F1 — the "armed margin" observed nothing, and used the wrong dtype's number

DevRelFor(dt) * 5 < kRopeSeparation compared two compile-time constants, so it measured nothing about the running system. And 2.11e-2 was the f32 arm's separation. Measured live, per dtype, on this fixture (Thor sm_110):

arm	a rotation moves this block by	band	ratio
f32	2.10893e-2	1e-5	2109x
bf16	1.97151e-2	4e-3	4.93x

So the stored constant was ~6% too large for bf16 — and once the real number is read, the invented * 5 factor reds a bf16 arm that is in fact correctly armed. Lowering 5 to 2 to get green would be widening a scope to turn a gate green, which is the one thing AGENTS.md forbids outright.

The proxy is gone. The case builds the rotated counterfactual from the same per-dtype fixture (RefAttentionRotated, which carries its own q_moved instrument) and asserts the property: run the rotated answer through the same ExpectCloseRel arithmetic that accepted the real one, and require it to come out rejected. No stored twin, no safety factor, measured per dtype per run. The host no-RoPE case shares the helper, deleting 40 lines of inlined attention core.

F2 — the residency case gated a test-local copy, and nk had no reader at all

The fixture builds its OwnedTensors with the test's own OwnT, not with production CopyDenseOwned. It can prove the device arm consumes the type; it can never see the loader mis-produce it. Both halves are now closed against the real checkpoint.

host_bytes was printed and read by nothing. Truncating every production payload to half moved it 470 MiB with the forward suite 16/16 green and 3/3 oracle goldens still MATCHING — a shrunk std::vector keeps its buffer, so View() reads the same values back and only the accounting moved. Both totals are now asserted exactly, against the content-pinned revision.

CHECK(rep.host_bytes == rep.source_bytes) is deliberately NOT in the file, because it is false. The review proposed it on the strength of the printed line host bytes: 18013 MiB, source 18013 MiB, but MiB resolution hides the difference: measured exactly, they differ by 15324 bytes, so that assertion would have redded on correct code. What is asserted instead is both exact totals and the identity that explains the gap, so the number is accounted for rather than tolerated.

	bytes
6039 f32 scalars READ but never stored as payload (5935 NVFP4 weight_scale_2 + 46×2 FP8 + 12 fp8-KV)	−24156
the 69 f32-by-contract SSM widenings (23 × 3 × 64 heads × 2 B)	+8832
host_bytes − source_bytes	−15324

nk had no consumer on this path, exactly as the review found: View() does not propagate it and both arms called vt::MatmulBT unconditionally. Rather than delete the intent, it now has a real reader in the tree's own idiom (qwen3_5.cpp:3353, :3611, :7638): the two functions that call MatmulBT — nemotron_h.cpp:317 (host) and nemotron_h_device.cpp:261 (device) — refuse a weight whose recorded orientation is not the [out, in] one MatmulBT reads. The tautological fixture assertions are deleted.

F3 — the GB10 leg is RUN, not owed: the bands transfer unchanged

The review asked for this to be documented as owed, because dgx's gpu.lock was held. It came free, so the leg was run instead, and F3 is closed by measurement rather than by a note.

The concern was specific and correct: the bands were calibrated against Thor's fallback kernels, with fa2 and cutlass-* reporting DISABLED for [110]. On GB10 the same device arm runs FA2 + CUTLASS GEMMs against the same CPU host reference, and the f32 band 1e-5 sat only 4.4x above Thor's measured 2.29e-6. A different attention kernel could plausibly have eaten that.

It did not. The separation is identical to Thor's, to six figures.

	Thor sm_110 (fallback)	GB10 sm_121a (FA2 + CUTLASS)
f32 device-vs-host separation	2.29122e-06	2.29122e-06
bf16 device-vs-host separation	0 (bit-exact)	0 (bit-exact)
f32 band DevRelFor	1e-5	unchanged, 4.4x headroom
bf16 band DevRelFor	4e-3	unchanged
rope separation f32 / bf16	2.10893e-2 / 1.97151e-2	identical

The rope separations matching exactly is expected and is a property of the repair: the counterfactual is built from the fixture in double precision on the host, so it is host-independent by construction. The device-vs-host separations matching is the actual finding — this arm's ops (vt::Attention, MatmulBT, RmsNorm, Embedding, FusedChain) land on the same answer through both kernel families.

The build is the non-degraded one, which is what makes the result admissible. A degraded build here would answer the Thor question twice rather than the GB10 question once, so the script voids rather than fails on any missing fast path. Configure at 121a:

-- CUDA target architectures: 121a
--   CUDA feature fp4-mma: ENABLED for [121a]
--   CUDA feature cutlass-nvfp4: ENABLED for [121a]
--   CUDA feature cutlass-fp8: ENABLED for [121a]
--   CUDA feature marlin-nvfp4: ENABLED for [121a]
--   CUDA feature fa2: ENABLED for [121a]
-- CUTLASS found at /cutlass; enabling sm120a NVFP4 cutlass GEMM
-- Triton AOT: ... <- sm_121a as vt_aot_sm_121a_gdn_deltah_h48_default
VOID_FLAG=0

cutlass-fp8: ENABLED, so the result stands. BUILD_EXIT=0, compile_errors: 0, enospc_hits: 0, -j 4, 3.0 TB free. Binaries a330e72f… / 68eb1e88….

suite, GB10 sm_121a	cases	assertions	status
test_nemotron_h_forward	16 / 16	5716	SUCCESS
test_nemotron_h_scaffold	14 / 14	38302	SUCCESS
test_nemotron_h_loader (real checkpoint)	2 / 2	53	SUCCESS, 3/3 oracle goldens MATCH

host bytes exact: 18888922112, source bytes exact: 18888937436 — byte-for-byte the Thor values, which is the evidence that the two new exact literals are properties of the pinned checkpoint plus the loader and not of a host.

Two honest caveats on this leg. local-ai-worker was not parked: there is no systemd unit by that name on dgx in its current state, so systemctl stop was a no-op and the ./local-ai worker process ran throughout. That is inert for this measurement — every number here is a numerical separation, not a timing — and it would have voided a throughput result, which this row does not record. Free memory during the run was 114 GiB of 119, so there was no pressure. The gpu.lock was held for the whole run via flock and is released.

F5 — "both arms share one reader" is not literally true

nemotron_h.cpp:56 and nemotron_h_device.cpp:126 are two file-local function-local statics with byte-identical predicates, latching independently. Nothing straddles today and the duplication is the tree's idiom (qwen3_5.cpp:1699), but the comment claimed impossibility. It now says what is true: a convention, not an impossibility, and hoisting the predicate is a tree-wide change rather than this row's.

F7 — one dead device allocation per layer per step

DBuf mixer_out(d, adt, {T, H}) was allocated and then move-assigned over on both branches. The mixer output now goes straight into carry, which is the only thing that reads it; the previous carry block returns to the pool at exactly the same statement as before, so the lifetimes are unchanged. No speed claim is made or implied.

Gate

Thor (sm_110), FRESH build directory, real checkpoint

Both gate hosts were exercised — Thor below, GB10 in F3 above. Spec §5.4 requires both, and Thor is the portable/fallback path rather than a second CUDA host.

COMPILE_EXIT=0   compile_errors: 0   warnings: 0   enospc_hits: 0

suite	cases	assertions	status
test_nemotron_h_forward	16 / 16	5716	SUCCESS
test_nemotron_h_scaffold	14 / 14	38302	SUCCESS
test_nemotron_h_loader (real 20.1 GiB checkpoint)	2 / 2	53	SUCCESS, 3/3 oracle goldens MATCH

Baseline binaries: test_nemotron_h_forward 34a49eaf…, test_nemotron_h_loader 4cfeb7e9…. Device confirmed in-container (NVIDIA Thor), no skip message emitted. Per case, each with a non-zero case count:

case	cases	assertions
device-consumed weights are in the shared residency type	1	5408
device attention block matches the host reference block for block	1	34
hybrid device forward matches the host reference token for token	1	20

The gate was re-run at the pushed head, not only at the head the mutations used. The final commit is comment-only, and the rebuilt binaries hash to 34a49eaf… / 4cfeb7e9… — byte-identical to the baseline — so the whole mutation table below is valid at the merged head rather than at a superseded one.

Resolved checkpoint directory recorded as evidence: /w/ckpt/nemotron-3.5-lightning-30b-nvfp4, VT_NEMOTRON35_SNAPSHOT unset. host bytes exact: 18888922112, source bytes exact: 18888937436. Thor disk 338 GiB free before and after.

Mutations

Each applied alone to a scratch copy with anchor uniqueness (count == 1) asserted on every edit, rebuilt, run, then restored — and every restore rebuilds back to the baseline binary sha, which is the control that proves the tree came back byte-for-byte.

#	mutation	compile	mutated sha	result
M2	the device block applies the NeoX rotation this architecture does not have	0, 0 errors	fwd b8394994…	RED BOTH DTYPES. attention case 34 assertions / 5 failed (3 at f32, 2 at bf16); hybrid 20 / 4 failed. bf16 device-vs-host separation 0.0200768 against band 0.004
M3	revert to the old RelFor bands with both new guards disabled, plus M2	0, 0 errors	fwd 4a42e062…	RED f32 ONLY in the attention case: 34 / 2 failed, both f32; bf16 GREEN. The historical hole, reproduced
M3B	widen the bands back to RelFor, no product mutation at all	0, 0 errors	fwd 0c30f4e1…	RED bf16 ONLY: 34 / 2 failed, both bf16; f32 GREEN. The new guard alone closes the hole, on correct code
M4	the loader records the wrong nk for all four attention projections	0, 0 errors	fwd e22fb03f…, loader ea0f360b…	forward 16/16 GREEN (the synthetic fixture is blind, as the review said); loader 2 cases / 1 FAILED, THREW weight 'mixer.q_proj' is not in the [out, in] torch-Linear orientation vt::MatmulBT consumes
M5	production CopyDenseOwned truncates every payload to half	0, 0 errors	fwd 5f113cce…, loader 3771a85f…	forward 16/16 GREEN and 3/3 goldens still MATCH; loader 53 / 2 failed on host_bytes == 18888922112 (read 18396080000) and on the composition identity
M3, first attempt	—	COMPILE_EXIT=1, 1 error (unused variable 'atol')	fwd b8394994… — identical to M2's, i.e. unchanged	VOID, not a result. Reported rather than counted: it would have read as a pass

Two traps caught by the harness rather than by luck, both worth recording:

• A mutation that fails to build reads as a passing test. The first M3 attempt did not compile, and the binary it ran was M2's. Printing COMPILE_EXIT, the error count and the sha beside every result is what caught it.
• shutil.copy2 preserves mtime, and ninja keys on mtime. A restored file older than its object silently keeps the previous mutation's object in the binary — the first M3B run reported a rotated device arm on unmutated source. Fixed by stamping mtime on restore, and the restore-to-baseline-sha control now makes that class of error impossible to miss.

M4 and M5 are both caught only by the checkpoint-gated loader suite, because they are defects in how the loader produces a weight and no synthetic fixture runs that code. That is stated plainly rather than presented as CI coverage.

Full gate

ctest on the local x86_64 development arm, whole suite: 485 / 485 passed, 0 failed (2 skipped: test_modelopt_mixed_precision_checkpoint, test_voxtral_e2e), 488.96 s.

scripts/agent-preflight.sh: 1 gate failed — test_cpu_x86_llamacpp_floor — and it is not this change's, which was verified rather than assumed. The failure is NO_QUIET_WINDOW … exit 4, the harness refusing to measure, not a floor violation, on a box at load average 157 from this session's own 973-target build. Re-run at load average 15 on the same tree: Ran 10 tests in 49.154s / OK, exit 0. So it is a load artefact, and the local gate is green on a quiet box. This diff touches no CPU and no llama.cpp path. The inherited baseline was RE-MEASURED on this head rather than subtracted from a list. That matters, because it shrank: 4880c5715 — picked up by the pinned merge above — landed #904's LTX-2.5 device use-after-free, and sanitize-cpu is now GREEN on both legs (address,undefined 41m17s, thread 44m45s). It is therefore no longer inherited, and a red there would have been mine. The seven #873 gates are likewise green, repaired by #878, and are not subtracted.

That leaves exactly one inherited red on CI: windows-msvc-cpu and windows-msvc-vulkan (#584), which have no main baseline and so are red on every PR. Verified rather than assumed to be inherited — both were already failing on the reviewed head f77dbac3f (gh api .../commits/f77dbac3f.../check-runs → windows-msvc-cpu failure, windows-msvc-vulkan failure), and the failing step is Build and execute the native Windows CPU focused gate. Everything else on the PR passes, including cuda-fat-build (1h20m35s), build-test-cpu, build-test-cpu-arm64, build-test-vulkan, device-leakage, pr-size, agent-record, documentation-checkpoint and commit-protocol-tag.

origin/main was merged at the pinned SHA 4880c5715 per #841. Three files overlap — CMakeLists.txt, docs/FEATURES.md, docs/USAGE.md — and all three are keyed; git diff 4880c5715 after the merge is exactly this row's own edits and nothing else, so every other key came from the target byte-for-byte. The merge also caught the one file the rename had missed (CMakeLists.txt:882).

Docs: FEATURES.md and USAGE.md updated (both owed and both demanded by check-doc-checkpoint). No lifecycle change, so no STATUS/BENCHMARKS/NOW write is owed.

One trap this PR hit, recorded rather than worked around

commit-protocol-tag reported this body as missing Following-Agents-Protocol and AI-Assisted while it carried both, correctly, in a well-formed final paragraph. The cause was two bare --- markdown horizontal rules: squash_merge_commit_message = PR_BODY makes this body the landed commit message, and git interpret-trailers treats a bare --- as the end of a commit message, so everything below it — trailer block included — is never parsed. Measured both directions on the stored body: --parse returned nothing; sed 's/^---*$/***/' returned all three trailers. The checker was right and the body was wrong. Written up in .agents/style/commits.md so the next author is told which character did it. A table's |---|---| is unaffected — the line is not bare.

Follow-ups filed and referenced, not folded in: #940 (extract the shared FP8 W8A8 linear seam), #941 (the spec's §2 dense_attn::AttnBlock naming is wrong).

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]