NemotronH A3 on GB10 sm_121a reads 95/96 DIVERGENCE on BOTH recurrent decode arms; Thor sm_110 reads 96/96 on the same code

[localai-bot]

docs/STATUS.md has carried "sm_121a re-run pending" for this row. It has now been run, and it fails — on both recurrent decode arms, so it is not attributable to whichever arm is selected.

The measurement

dgx:gpu0 (GB10, sm_121a) inside an rc lease, tree e35c14d52 on row/A2-D1-mamba-decode-step, recipe scripts/nemotron-h-a2d1-gpu-gate.sh, real 20.1 GiB NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4, device mamba arm ON in both legs, cutlass-fp8: ENABLED for [121a] (5 feature cells enabled, so the build is not VOID).

	ON — single-step decode (vt::Mamba2StateUpdate)	OFF — chunked decode (vt::Mamba2ChunkScan)
RC[a3]	1	1
TOKEN MATCH	95/96 full rows=3, short rows=0, mode=decode	95/96 full rows=3, short rows=0, mode=decode
verdict	DIVERGENCE	DIVERGENCE
decode-step counters	state_update_rows=23 chunk_scan_calls=0 gathers=0 scatters=0	state_update_rows=0 chunk_scan_calls=23 gathers=46 scatters=46
reference-tier lines	0	0
per output token	1.513958 s	1.544706 s
GPU busy, decode samples	112 of 1013 = 11.06%	120 of 1051 = 11.42%

The counters confirm the two legs really did run different kernels, and both lose exactly one token in 96.

Why this is not the decode-arm change

Same binary, same box, same checkpoint, same prompts. The arm that predates #1311 diverges identically to the one that replaces it, so #1311 is neutral to this. The same code passes on other silicon:

host	ON	OFF
thor:gpu0 sm_110	96/96 STRICT PASS	96/96 STRICT PASS
dgx:gpu0 sm_121a	95/96 DIVERGENCE	95/96 DIVERGENCE

So it is arch- or host-specific, not arm-specific.

What is NOT established

Whether both legs lose the SAME token. The counts and the row shape (full rows=3, short rows=0) are identical, which is what supports "neutral", but the driver's got: / exp: ids were discarded by the gate script's verdict grep and are not in the run log. That was a defect in the recipe and is fixed on the branch, so the next run captures them. Until then, "one token in 96 on both arms" is the claim, not "the same token".

That also means a wrong recurrent carry and a benign bf16 near-tie are not yet separated. A single divergent token out of 96 looks identical either way.

Related, and probably not independent

• docs/STATUS.md: "GB10 read 4/24; cause and fix NemotronH A3 token gate DIVERGES on real weights: token 0 is correct on all 3 prompts, then decode collapses to a repeated token (6/96) #1157, sm_121a re-run pending" — this IS that re-run.
• NemotronH: the HOST mamba arm reads 93/96 on Thor sm_110 and 96/96 on GB10 sm_121a, against the same golden #1290 records a 93/96 DIVERGENCE arm for A2-Q1 and concluded the host-arm divergence was an sm_110 result with GB10 as a counter-measurement. This run is the GB10 side and it is not clean.

Next traceable step

Re-run the gate on dgx:gpu0 from a branch carrying the repaired script and read the got: / exp: ids for the mismatching row: that says which prompt and which position, and whether the two arms lose the same one. Only then is it worth asking whether the oracle's top-2 margin at that position makes it a tie.

Filed rather than fixed in flow: the defect is in NemotronH on GB10 and predates the change that found it, and it needs its own spec and a lease rather than riding on #1311.

[localai-bot]

⚠️ Correction: "arch-specific, not arm-specific" is UNDER-DETERMINED

I framed this as the host's divergence. That conclusion is not supported by the A/B I ran, and my own counters are what show it. Correcting before anyone acts on it.

The confound

Both legs of the A/B varied the mamba kernel — single-step vs chunk scan — and both sat on top of A2-Q1's FP8 W8A8 device projections. The four kernel counters are non-zero in each leg, and they are reachable only from the mamba_on_device branch, so both legs entered it. The gate script pinned VT_NEMOTRON_H_DEVICE_MAMBA=1 in both.

main carries no device mamba arm at all:

$ git show origin/main:src/vllm/model_executor/models/nemotron_h_device.cpp | grep -c NemotronHMamba2MixerDevice   -> 0
$ git show origin/main:src/vllm/model_executor/models/nemotron_h_device.cpp | grep -c MambaIsFp8                   -> 0
$ git show origin/main:docs/ENVIRONMENT.md | grep -c VT_NEMOTRON_H_DEVICE_MAMBA                                    -> 0

So 95/96 on both legs is equally consistent with:

• (a) the divergence being the HOST's — arch-specific, as first written; or
• (b) the divergence being A2-Q1's FP8 projections (#1289), which neither leg turns off.

This A/B cannot separate them. Concluding (a) was reading a conclusion out of an experiment that did not test it.

A third data point favours (b) without settling it

A GB10 sm_121a A3 run on a main-based tree — therefore with no device mamba arm — read 96/96 STRICT PASS. Different branch and different binary, so it is corroboration, not the discriminator.

tree	device mamba arm	GB10 A3
main-based	absent	96/96 STRICT PASS
#1289-based, single-step leg	present	95/96 DIVERGENCE
#1289-based, chunk-scan leg	present	95/96 DIVERGENCE

The discriminator, queued

VT_NEMOTRON_H_DEVICE_MAMBA=0 on the same binary and box, routing the whole mamba block back to the host reference — the only difference. Its own counters check it took that path: the four kernel counters must read 0 while gathers/scatters stay non-zero, because the host branch still gathers and scatters but never enters the instrumented device mixer. A leg that cannot prove which path it took discriminates nothing.

• leg 3 96/96 ⇒ the divergence is A2-Q1's FP8 arm, not the architecture. This issue gets re-scoped and feat(#810 A2-Q1): NemotronH's 23 Mamba2 blocks reach the device on the FP8 W8A8 seam (#517) #1289 needs flagging — it is the PR carrying the only real speed win measured this session (2.31x occupancy, 7.17x per-token on Thor).
• leg 3 95/96 ⇒ the host owns it and the original framing stands, strengthened.

The same run also carries the repaired verdict grep, so it captures the driver's got:/exp: ids. With those, "same token lost on both legs" becomes checkable instead of the weaker "matching counts and row shape", and the oracle's top-2 margin at that position can be asked — this family has already produced one "divergence" that was a bit-exact near-tie, which would also explain why Thor's different attention arithmetic does not show it.

BENCHMARKS, STATUS, the row spec and the benchmark record now all say the cause is OPEN rather than asserting the host.

[localai-bot]

The three-leg discriminator ran on dgx:gpu0 and the cause is settled: this is A2-Q1's device mamba arm, not the architecture. Logs: /workspace/a2d1-discriminate/20260819T200231Z.

The result

leg	VT_NEMOTRON_H_DEVICE_MAMBA	kernels	tokens	s/token	vs vLLM
on	1	single-step	95/96 DIVERGENCE	—	—
off	1	chunk scan	95/96 DIVERGENCE	1.554	108.2x
hostmamba	0	none (host)	96/96 STRICT PASS	10.319	718.1x

One binary, one box, one checkpoint. The only variable across leg 3 is whether the device mamba arm runs at all.

The falsifier PASSED, so this is a result and not a coincidence

Leg 3 had to prove it actually took the host path, or the discrimination would have been VOID:

[NH-DIAG] ARM step T=1 nd=1 np=0  state_update_rows=0 chunk_scan_calls=0 \
          conv_update_rows=0 conv_fwd_calls=0 gathers=46 scatters=46

All four kernel counters 0, gathers and scatters still 46 — the host branch gathers and scatters but never enters the instrumented mixer. That is exactly the required shape. 96 ARM lines, 93 decode rows, 3 prefill rows.

This also retires the confound. The earlier on/off A/B could not separate architecture from arm because BOTH legs ran A2-Q1's FP8 projections. Leg 3 is the first run that turns the arm off, and it is the leg that passes.

What actually diverges

One token, and it is the last one, on the longest prompt:

prompt 0: prompt_tokens=5   compared=32 matched=32
prompt 1: prompt_tokens=8   compared=32 matched=32
prompt 2: prompt_tokens=13  compared=32 matched=31
  got: ...,1044,12837,2505,1261,9943,1307,11286
  exp: ...,1044,12837,2505,1261,9943,1307,3468

Positions 1-31 are identical. Only position 32 differs.

Hypothesis, stated as one and not as a finding: the FP8 W8A8 mamba projections accumulate rounding error in the CARRIED conv + SSM state across decode steps, so the drift only crosses a decision boundary at the last step of the longest sequence. That predicts the divergence point moves with sequence length and disappears at short generations. It is testable and unconfirmed.

The cheaper question first, and it is still unasked: the oracle's top-2 margin at that token. This family has already produced a "divergence" that was a bit-exact near-tie. If the margin is at the noise floor this is a tie-break, not a defect. Ask that before building anything.

Re-scoping

The title's claim -- "on BOTH recurrent arms" -- is true but misleading, because both arms shared the FP8 projections. This issue is now: A2-Q1's device mamba arm loses one token on sm_121a. The architecture-level framing is withdrawn.

main is unaffected: it has no device mamba arm (MambaIsFp8 count on origin/main = 0) and reads 96/96 on this box.

[localai-bot]

The oracle is not token-stable against itself on this prompt

STEP ONE asked for the oracle's top-2 margin at the divergent token. While that
measurement was queued on dgx:gpu0, four things already in the record answer
most of the question, and one of them is decisive on its own.

1. The token, decoded

Prompt 2 generates, through position 29:

" A state space model is a mathematical framework that describes a system's
evolution over time through a set of hidden states and observed outputs,
typically using a"

The three arms then continue differently, and every one of them is fluent:

arm	positions 30, 31, 32	reads
vLLM golden	9943 combination, 1307 of, 3468 state	"…using a combination of state"
Thor sm_110, HOST arm (W8A16)	11286 transition, 12927 equation, 1394 for	"…using a transition equation for"
GB10 sm_121a, DEVICE arm (W8A8)	9943 combination, 1307 of, 11286 transition	"…using a combination of transition"

The competing token is the SAME one — 11286 transition — in both divergent
arms, at position 30 in one and 32 in the other. All three tails are the same
sentence taking a different fork.

2. The two legs that diverge run DIFFERENT kernels and lose the SAME token

/workspace/a2d1-discriminate/20260819T200231Z, one binary, one box, one
checkpoint:

leg	recurrent kernels	decode-step counters	prompt 2
on	vt::Mamba2StateUpdate	state_update_rows=23 chunk_scan_calls=0 gathers=0 scatters=0	matched=31, got …,9943,1307,11286
off	vt::Mamba2ChunkScan	state_update_rows=0 chunk_scan_calls=23 gathers=46 scatters=46	matched=31, got …,9943,1307,11286

Two different recurrent kernels, and a gather/scatter count that differs by 46,
produce a byte-identical wrong token. That excludes the recurrent kernel's
reduction order and the gather/scatter state indexing as the cause: a defect in
either could not be invariant across that pair.

3. The polarity FLIPS with silicon, and the MORE PRECISE arm is the worse tracker

Same source, same golden, same staged checkpoint (/workspace/a3/ckpt-stage),
same driver:

	HOST arm — W8A16 (dequant to bf16, input_scale carried and NOT applied)	DEVICE arm — W8A8 (what vLLM does)
thor:gpu0 sm_110	93/96 (2 independent leases agree)	96/96
dgx:gpu0 sm_121a	96/96	95/96

Two readings follow, and they point the same way:

• A real FP8 numerics defect in the device arm cannot be 96/96 on Thor. The
  arm is not systematically wrong; it is silicon-sensitive.
• The host arm is strictly more precise than vLLM itself — it never
  quantizes the activation. If precision loss drove this, that arm would be the
  best tracker of a vLLM golden. On Thor it is the worst, and it diverges two
  positions EARLIER. That ordering is the opposite of what accumulating
  quantization error predicts.

4. THE DECISIVE ONE: vLLM disagrees with vLLM on this prompt, by SIX tokens

/workspace/nhspeed/oracle.a.out, 2026-08-18. The pinned oracle
(0.23.1rc1.dev1511+g555967922, FLASHINFER, CUDA graphs ON) re-ran its OWN
committed golden on this box, on the same staged checkpoint, same prompts, same
greedy sampling — changing only the engine configuration (max_model_len=512,
max_num_batched_tokens=512, gpu_memory_utilization=0.3):

ORACLE prompt 0: prompt_tokens=5  compared=32 matched=32
ORACLE prompt 1: prompt_tokens=8  compared=32 matched=32
ORACLE prompt 2: prompt_tokens=13 compared=32 matched=26

Reproduced identically in a second leg of the same run.

Prompts 0 and 1 are bit-stable across the configuration change. Prompt 2 moves
by six tokens. Our device arm moves by one.

A token-exact gate on prompt 2 is therefore a bar the oracle does not meet
against itself, and 95/96 on that prompt is inside the oracle's own
configuration-to-configuration spread rather than outside it.

What this does NOT establish

The top-2 margin in nats at position 32 is still the number that turns this from
a strong inference into a measurement, and it is what decides the FORM of the
replacement gate. The instrument is staged and queued.

[localai-bot]

Two data points from #1250's speed measurement that this issue does not have, and the first one narrows its conclusion.

The HOST mamba arm PASSES 96/96 on GB10 sm_121a. This issue reasons "arch- or host-specific, not arm-specific" from two DEVICE arms plus a passing Thor, but it had no host-arm leg on GB10 to compare against. There are now three:

run	arm	GB10 result
#1250 timing leg 1	host bounce	96/96 STRICT PASS
#1250 timing leg 2	host bounce	96/96 STRICT PASS
/workspace/a2d1-discriminate/20260819T200231Z a3_hostmamba	host bounce	96/96 STRICT PASS
same run, a3_off	device Mamba2ChunkScan	95/96 DIVERGENCE
same run, a3_on	device Mamba2StateUpdate	95/96 DIVERGENCE

The last three are one binary, one box, one checkpoint, with the arm as the only variable. On GB10 the host arm passes and both device arms lose one token, so the divergence does track the arm. What your Thor/GB10 table establishes is that the device arms are additionally arch-sensitive; it does not establish that the host arm shares the defect, and on this evidence it does not.

The diverging row is prompt 2, and the ORACLE ITSELF diverges on prompt 2. Both device legs lose their token on prompt 2 (31/32). Separately, in #1250's oracle leg the pinned oracle failed to reproduce its own committed golden on prompt 2 (matched=26 of 32) once its resolved block_size moved from the capture-time value to 512 under gpu_memory_utilization=0.30, max_num_batched_tokens=512. Same prompt, two unrelated perturbations, one of them inside vLLM itself.

That is evidence for the benign-near-tie branch of your "wrong carry vs near-tie" question and against a wrong recurrent carry, because a wrong carry would not explain the oracle diverging from its own golden under nothing but a block-size change. It also names the discriminator you already ask for: the oracle's top-2 margin at that position. Treat it as a lead rather than a finding until the got:/exp: ids confirm the two arms lose the same position.

Recorded in .agents/benchmark-record.md under MODEL-NEMOTRON-H-ABI-A2P on PR #1251.