feat(#810 A2-Q1): NemotronH's 23 Mamba2 blocks reach the device on the FP8 W8A8 seam (#517)

[localai-bot]

feat(#810 A2-Q1): NemotronH's 23 Mamba2 blocks reach the device on the FP8 W8A8 seam (#517)

The 23 Mamba2 layers were the last host bounce of a decode step, and they were
the expensive one. The host arm reaches its two projections through
Linear(..., const NemotronHOwned&), whose DenseFor calls
NemotronHOwned::DenseBf16() -- a FULL dequant of the fp8 tower into a fresh
bf16 buffer ON EVERY CALL. That is 23 x (10304x2688 + 2688x4096) = 890e6
elements re-expanded per token, plus a download of the normed hidden and an
upload of the mixer output per layer.

The block moves whole or not at all: mixer.in_proj produces the fused zxbcdt
that the causal conv and the SSD scan both consume (mamba_mixer2.py:550, split
:692-696), so there is no intermediate landing in which the conv is on the
device and in_proj is not. That is why the shared FP8 W8A8 linear seam had to
be extracted first (#940, dense_fp8_gemm.h), and this change routes through it
rather than re-typing its entry points.

What it does

NemotronHMamba2MixerDevice mirrors the host arm statement for statement --
same vt:: ops, same order, same dtypes, different backend:

in_proj (FP8 W8A8) -> QkvSplit(z | xBC | dt) -> CausalConv1dFwd(silu)
  -> QkvSplit(x | B | C) -> Mamba2ChunkScan -> RmsNormGatedGroup(n_groups)
  -> out_proj (FP8 W8A8)

The one substitution is the split: the host arm copies columns with SliceCols
because vt::Mamba2ChunkScan validates every operand contiguous, and
vt::QkvSplit is exactly that copy on the device.

Both forwards select it at runtime, never by preprocessor. MambaIsFp8 names
only NemotronHWeightForm and vt::OpRegistered is the op table's own answer,
so a dense NemotronH or a device without the fp8 pair keeps the host bounce.
The paged arm hands the mixer the recurrent rows vt::GdnStateGather already
gathered and zeroed and it advances them IN PLACE, so the two downloads, the
host mixer and the two uploads per layer are gone; A2-P's gather/scatter pair
and its has_initial = true reasoning are unchanged.

The ssm_dtype == f32 term in the paged selection is not decoration. The gather
widens the page into an f32 working buffer by op contract and the host arm then
narrows it back to ssm_dtype, so on a checkpoint whose mamba_ssm_cache_dtype
is not f32 the two arms would round differently and the per-block numeric gate
would compare two different computations.

Three points where this differs from the spec, deliberately

1. The Fp8Weight pair is built ON FIRST DEVICE USE in a ResidentSlot the
   weights own, not by the loader as spec §4.1/§4.2 proposed. That is A2-Q2a's
   newer idiom and it is better here: it does not double the 890 MB tower in
   host memory at load, it uploads nothing on a host-only run, and it leaves
   rep.host_bytes (the literal 18888922112 pinned at
   test_nemotron_h_loader.cpp:310) untouched, so §4.2's re-derivation
   obligation does not arise. Keying on the slot rather than an address is
   CUDA BF16 MoE resident cache reuses stale pointers after engine teardown #237: across two engine builds in one process an address-keyed cache returns
   the previous engine's device pointers, which is plausible wrong values.
2. The upload IS accounted, where §4.4 expected the report to say it was short.
   dense_fp8::ResidentFp8 still does not call load_stats::AddDeviceUpload --
   that is FP8 W8A8 weights are uploaded without load accounting or the post-upload residency step, unlike every other resident weight in the same file #974, unchanged, and the shared header is not touched -- so A2-Q1
   accounts what IT uploads at the site that causes it, as ResidentWeight and
   ResidentNvfp4 do.
3. The e4m3 staging copy is released as soon as ResidentFp8 has uploaded it,
   so the conversion peaks at one projection rather than at the whole tower.

The comparison is not bit-comparable, and that is the point

The host reference is W8A16: DenseBf16 says outright that input_scale is
carried and not applied. The device arm is W8A8, which is what vLLM does. The
difference between them is the e4m3 activation quantization, so every band in
tests/vllm/models/test_nemotron_h_mamba_device.cpp is measured in the run
against a defect the fixture separates, and every element count is asserted
against the geometry -- a maximum over zero elements is also 0.0.

The new file is the cheap arm in front of the real-checkpoint gate, for the same
reason A2-Q2a's device MoE file exists: BuildTiny is all-kDense, so nothing
in the existing device suite could reach the FP8 W8A8 arm at all.

G-SAFE

Untouched. All three clauses of the interlock at
nemotron_h_registry.cpp:162 stand. This change creates no paging, no carried
state and no batching that A2-P did not already create, and num_reqs <= 1
remains A2-B's to remove.

What is NOT proved here

The §5.1 per-block numeric gate on the real 20.1 GiB checkpoint, the A3
96/96 mode=decode STRICT PASS re-run, the §5.3 mutation pass and the GPU busy
fraction are the unit's acceptance test and they need a GB10 lease.
scripts/nemotron-h-a2q1-dgx-gate.sh records the recipe, including the three
environment facts that have each voided a run before: the CUDA lane is sbsa,
nvcc --version is not a sufficient postcondition, and /workspace is CIFS and
refuses symlinks. The spec's ## Now and ## Owed say so, and
docs/BENCHMARKS.md records the measurement as PENDING rather than as silence.

Issue: #810 (parent row #517). Spec:
.agents/specs/nemotron-h-a2q1-fp8-mamba.md.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

[localai-bot]

Superseded by #1289, which carries the identical tree on a linear branch off origin/main.

This branch's merge commit 49f3515f3 was created with git's default Merge remote-tracking branch message, so it carried neither the bare FOLLOWING_AGENTS_PROTOCOL paragraph nor the trailer block, and scripts/check-commit-trailers.py failed on it. A merge commit here is a commit like any other and owes the same message; the repository's own merges on main carry it.

Closed rather than force-pushed: AGENTS.md says never force-push and never add a force option, without carving out task branches. #1289 is the same change with one commit off the current origin/main, so no merge commit exists to repair.