feat(#810 A2-Q1): NemotronH's 23 Mamba2 blocks reach the device on the FP8 W8A8 seam (#517) - #1289
feat(#810 A2-Q1): NemotronH's 23 Mamba2 blocks reach the device on the FP8 W8A8 seam (#517)#1289localai-bot wants to merge 18 commits into
Conversation
…e FP8 W8A8 seam (#517) The 23 Mamba2 layers were the last host bounce of a decode step, and they were the expensive one. The host arm reaches its two projections through `Linear(..., const NemotronHOwned&)`, whose `DenseFor` calls `NemotronHOwned::DenseBf16()` -- a FULL dequant of the fp8 tower into a fresh bf16 buffer ON EVERY CALL. That is 23 x (10304x2688 + 2688x4096) = 890e6 elements re-expanded per token, plus a download of the normed hidden and an upload of the mixer output per layer. The block moves whole or not at all: `mixer.in_proj` produces the fused `zxbcdt` that the causal conv and the SSD scan both consume (mamba_mixer2.py:550, split :692-696), so there is no intermediate landing in which the conv is on the device and `in_proj` is not. That is why the shared FP8 W8A8 linear seam had to be extracted first (#940, `dense_fp8_gemm.h`), and this change routes through it rather than re-typing its entry points. `NemotronHMamba2MixerDevice` mirrors the host arm statement for statement -- same vt:: ops, same order, same dtypes, different backend, which is the property A2-R established and the numeric gate reads: in_proj (FP8 W8A8) -> QkvSplit(z | xBC | dt) -> CausalConv1dFwd(silu) -> QkvSplit(x | B | C) -> Mamba2ChunkScan -> RmsNormGatedGroup(n_groups) -> out_proj (FP8 W8A8) The one substitution is the split: the host arm copies columns with `SliceCols` because `vt::Mamba2ChunkScan` validates every operand contiguous, and `vt::QkvSplit` is exactly that copy on the device. Both forwards select it at runtime, never by preprocessor: `MambaIsFp8` names only `NemotronHWeightForm` and `vt::OpRegistered` is the op table's own answer, so a dense NemotronH or a device without the fp8 pair keeps the host bounce. The paged arm hands the mixer the recurrent rows `vt::GdnStateGather` already gathered and zeroed and it advances them IN PLACE, so the two downloads, the host mixer and the two uploads per layer are gone; A2-P's gather/scatter pair and its `has_initial = true` reasoning are unchanged. The `ssm_dtype == f32` term in the paged selection is not decoration. The gather widens the page into an f32 working buffer by op contract and the host arm then narrows it back to `ssm_dtype`, so on a checkpoint whose `mamba_ssm_cache_dtype` is not f32 the two arms would round differently and the per-block numeric gate would be comparing two different computations. The device residency is built on first use and keyed on a `ResidentSlot` the weights own, never on an address (#237): across two engine builds in one process an address-keyed cache returns the previous engine's device pointers, which is plausible wrong values rather than a crash. The e4m3 staging copy is released as soon as `ResidentFp8` has uploaded it, so the conversion peaks at one projection rather than at the whole 890 MB tower. `dense_fp8::ResidentFp8` still does not account its own upload -- that is #974, and this change does not touch the shared header -- so A2-Q1 accounts what IT uploads at the site that causes it, through the same `load_stats::AddDeviceUpload` `ResidentWeight` and `ResidentNvfp4` call. That counter is also the instrument the new residency case reads: an arm that re-uploaded the tower every step returns identical numbers to one that uploads it once. G-SAFE is untouched. All three clauses of the interlock stand, this change creates no paging, no carried state and no batching that A2-P did not already create, and `num_reqs <= 1` remains A2-B's to remove. `tests/vllm/models/test_nemotron_h_mamba_device.cpp` is the cheap arm in front of the real-checkpoint gate, for the same reason A2-Q2a's device MoE file exists: `BuildTiny` is all-`kDense`, so nothing in the existing device suite could reach the FP8 W8A8 arm at all. The two arms are NOT bit-comparable by construction -- the host reference is W8A16 and the device arm is W8A8 as vLLM is -- so every band is measured in the run against a defect the fixture separates, and the element count is asserted against the geometry so a maximum over zero elements cannot read as agreement. `scripts/nemotron-h-a2q1-dgx-gate.sh` records the GB10 recipe rather than leaving it to be retyped, including the three environment facts that have each voided a run before: the CUDA lane is `sbsa`, `nvcc --version` is not a sufficient postcondition, and `/workspace` is CIFS and refuses symlinks. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
|
Evidence status, so nobody reads silence as a result. Two leases are out for the gates this unit is accepted on. Both build the PR's exact tree from a branch, so the recipe is the committed
Thor is the interesting half of the correctness question and not a consolation prize: seven of its eight CUDA feature cells resolve DISABLED for The GB10 leg is the binding one. Its acceptance test is not a ratio: the GPU busy fraction must RISE from the measured 6.31% baseline (0% in 2,019 of 2,155 samples), and the script reports the fraction WITH its sample count, because a fraction with no denominator is not a measurement. If it does not rise, that is a real result about the hypothesis and it will be reported as one. Until both legs report, |
|
Handover: what has and has not been measured, at MEASURED, x86_64 CPU-only development box (a result from it is not an A2-Q1 result, and none of it touches the device arm):
NOT MEASURED, and each is a gap rather than an omission:
Leases outstanding: If the busy fraction does not rise when it is finally sampled, that refutes the hypothesis rather than the change, and the next traceable candidate is already named in the brief and in |
…ms it divided, or refuses `examples/nemotron_h_gen` prints neither a rate nor a duration. It prints the engine load time and a `TOKEN MATCH: m/n` line, so the per-output-token number this row is measured against has to be derived from the wall clock, and a derived number that hides its terms is exactly how a rate over an unknown denominator comes to be quoted as if it had been measured. `scripts/nemotron-h-a2q1-per-token.py` prints the wall time, the load it subtracts and the token count it divides by on separate lines, and it refuses in the two cases where a plausible zero would otherwise be printed: when either term is absent from the log, and when the wall clock is not greater than the load it contains. The second case is not hypothetical -- the terms coming from different runs yields a NEGATIVE rate, which still formats like a measurement. The gate script calls it once per arm, so the device-on and device-off runs report the same shape. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…e it had to accept, so it gates the STATE Thor (sm_110) ran the focused suite and one case failed. The device arm itself passed everything else -- 49 assertions where a GPU-less box runs 4, the fresh block matched the host reference at T=1/8/12 (0.164, 0.282, 0.309 against a band of 0.5), both refusals threw, and the fp8 tower uploaded 61760 B on the first call and 0 B on the second. The failure is the instrument. The carry case banded the SECOND LEG'S OUTPUT against the separation of a dropped carry. On Thor the second leg agreed to 0.705 while a dropped carry separated by only 0.205, so the derived band (0.102) sat BELOW the deviation a FRESH leg already shows on this fixture. The two arms are W8A8 against W8A16, so a fresh leg already disagrees by the e4m3 activation quantization and a second leg compounds that with the same disagreement propagated through the carried state. A defect whose separation is smaller than the noise the comparison must accept is not resolvable from that comparison, and widening the band until it passes is what the spec's §8.1 says to stop for. So the assertion moves to what the carry actually IS. A dropped carry hands the next leg zeros, so the separation between the advanced state and a zeroed one is 1.0 by construction -- about six times the noise floor, which this fixture can genuinely resolve. The conv window and the SSM state are now banded separately, each against its own zeroed twin, each with its element count asserted against the geometry. The noise floor is MEASURED in the run, at the same width, from a fresh device leg against a fresh host leg, and printed beside the separation. The second leg's output is still measured, and it carries an assertion only when the separation exceeds twice that floor; the condition is printed either way, so "no assertion here" is a stated measurement rather than a silent hole. What this cannot see -- a carry advanced but wrong by less than the band -- is written down, and the real-checkpoint per-block gate is named as the instrument for it. Two things the failure exposed beside the band. The fixture left `mamba_ssm_cache_dtype` unset and so resolved bf16, which is NOT the configuration the paged forward selects the device arm for (`ssm_dtype == f32`); it now says float32 as the released checkpoint does, and the case asserts it, because a state silently halved is invisible to every comparison. And the whole file was a skip on a GPU-less box, so a new CPU-runnable case pins the op contract the split depends on -- three outputs of widths (I, conv_dim, num_heads) taking the head, middle and tail of an in_proj row. That case states plainly that it does not pin the production call site, which only the device numeric case covers. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
left open The spec's §3 measured Thor (sm_110) as having HALF an fp8 arm: the GEMM present through the unconditionally registered `kMatmulFp8CublasLt`, the activation quant trapped in a CUTLASS-gated translation unit that no sm_110 build compiles. #991 moved the registration out. Whether that actually reaches this model's arm was never run, and the spec said the conclusion had inverted twice during scoping, so it is worth writing down that it now has an answer. It does. On a build whose feature table reads `ENABLED for [110]: 1 ; DISABLED cells: 7` -- only `marlin-nvfp4`, with `cutlass-fp8` and both `scaledmm-c3x` cells off -- the FP8 W8A8 mamba arm executes: 49 assertions where a GPU-less box reports 4, the fresh block matching the host reference at three widths, and the fp8 tower uploading exactly once. The section also records the carry gate's failure and its repair, because the failure is the more instructive half: the case banded a defect whose separation (0.205) was SMALLER than the noise the comparison had to accept (0.164 on a fresh leg), which is §5.2's lesson arriving from the direction the spec did not anticipate. And it states which host can answer the occupancy question at all -- the 6.31% baseline is a GB10 number, so a Thor busy fraction would measure different silicon and answer something else. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…side a Thor number The A3 gate ran on Thor and printed `GPU busy in 358 of 2336 samples = 15.33% busy (baseline 6.31%)`. The denominator is right and the sample count is there, but the parenthesis is wrong: 6.31% was measured on GB10, and printing it beside an sm_110 number invites a comparison across two different pieces of silicon, two different clocks and two different memory systems. It answers a different question than the one it appears to answer. That is the failure this repository already knows by name -- a number quoted often enough comes to be treated as measured -- and a gate script is exactly where it starts, because the script's output is what gets pasted into a report. The baseline is now quoted only when `ARCH` is `121a`. On any other arch the line says so and names the comparison that IS valid there: the same-binary `VT_NEMOTRON_H_DEVICE_MAMBA` ON/OFF A/B on that same box. All three arms are self-tested, including the empty-sample arm, which reports UNMEASURED rather than 0% -- a busy fraction over zero samples formats identically to a genuinely idle GPU. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
Thor (sm_110) leg: the A3 gate PASSES with the device arm onUnder an Correctness first, and it holds:
The build this ran on is the interesting part. Its feature table read The focused suite reported 49 assertions where a GPU-less box reports 4, the fp8 tower uploaded 61760 B on the first call and 0 B on the second, and the neighbouring suites were all green: forward 16/16 (5716), paged 12/12 (3256), loader 2/2, moe_device 2/2 (29), Two instrument defects this run exposed, both fixedThe busy fraction was printing The carry gate banded a defect smaller than the noise it had to accept (detail in Still owed
No per-token time came out of this run: the Thor build predates the timing helper and its script line needed |
…line it cannot be read against The A3 gate ran on Thor (sm_110) with the FP8 W8A8 mamba arm on the device and returned 96/96, mode=decode, STRICT PASS, exit 0, with zero reference-tier lines. That last number is the one worth recording beside the verdict: the portable reference tier is numerically CORRECT, so a pass obtained on it is invisible in every other figure, and its absence is the only thing that separates the two. The busy fraction was 358 of 2336 samples, 15.33%. Both surfaces say explicitly that this is NOT comparable to the 6.31% baseline the unit is accepted against, because that baseline was taken on GB10 and this is different silicon. Recording the number without that sentence would have been the more useful-looking and less true entry, and the GB10 leg is still queued. BENCHMARKS also records what the run did NOT produce: no per-token time, because the Thor build predates the timing helper and its script line needed `bc`, which the container lacks. STATUS moves from one passing gate to two and names which arm the second one exercised. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…ered the wrong question (#1290) The Thor A/B ran both arms of the same binary against the same checkpoint and golden, differing only by VT_NEMOTRON_H_DEVICE_MAMBA. Two things came out of it and only one of them is a number this row may keep. THE ARM IS TOKEN-EXACT WHERE THE HOST ARM IS NOT. With the device arm on, the A3 gate read 96/96 mode=decode STRICT PASS, exit 0. With it off -- which is what main does on a GPU today -- it read 93/96 DIVERGENCE, exit 1. Zero reference-tier lines on both, so neither run took the portable tier. The mechanism is named rather than guessed: the golden comes from an oracle that computes these projections W8A8, and `DenseBf16` states outright at nemotron_h.cpp:419-422 that the host arm carries `input_scale` and does not apply it. That is #1290, filed while landing this row and fixed by it. It is n=1 per arm, and the row says so. THE BUSY FRACTION IS VOID, AND IT IS THIS UNIT'S ACCEPTANCE TEST. It read 15.33% on and 14.73% off, and neither figure means what it appears to: the sampler started with the process, so the multi-minute GPU-IDLE 20.1 GiB engine load sat inside the same window as the decode and diluted both arms toward each other. That is the same defect as summing prefill and decode into one profile, and a fraction over the wrong window is worse than no fraction, because it still formats like a measurement. No occupancy claim is made from that run in either direction. `run_gate` now starts the driver first, waits for `engine loaded in Ns`, and samples only after it; when that line never appears it reports the fraction as NOT REPORTED rather than sampling a window whose meaning is unknown. The per-token helper takes the decode window directly and subtracts nothing, since the load is already outside its brackets, and it refuses on a non-positive window -- the brackets failing to span a decode would otherwise print a negative or infinite rate that still looks like a measurement. What survives from the run without an instrument caveat is the wall-time ratio from the sample counts, one sampler at one nominal interval: 2.44x shorter with the arm on. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
`origin/main` moved while this row's Thor A/B was running, so the branch is merged rather than rebased and the gate is re-run over the merged tree. AGENTS.md R4: merge an immutable SHA and re-run the full gate after merging rather than reading the diff. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
The A/B is in, and it says two things — one of which is that this unit's headline number is VOIDSame lease on
The host arm is the one that diverges
★ The GPU busy fraction is VOID, and it is this unit's acceptance testIt read 15.33% on and 14.73% off. Neither figure means what it appears to. The sampler started with the process, so the multi-minute, GPU-idle 20.1 GiB engine load sat inside the same window as the decode and diluted both arms toward each other and toward zero. That is the same defect as summing prefill and decode into one profile, and a fraction over the wrong window is worse than no fraction because it still formats like a measurement. So no occupancy claim is made from this run, in either direction. The hypothesis that this arm raises GPU occupancy is neither supported nor refuted by it. What survives without an instrument caveatFrom the sample counts, one sampler at one nominal interval: the run was 2.44x shorter with the arm on (2336 vs 5702 samples). The GB10 leg is still queued and remains the only one that can answer the 6.31% question — now with a sampler that measures the right window. |
…a counter-measurement The issue as filed said the host mamba arm is what main does "on a GPU today" and therefore that main is token-wrong on a GPU. That is established for sm_110 ONLY. On GB10 (sm_121a) the SAME host arm reads 96/96 STRICT PASS -- the A3 run that closed #1157, /usr/local/nas_share/rc/nh1157/gate_fixed.out, same golden and same oracle revision. Left as written, the issue would get dismissed the first time somebody checked on GB10 and got a pass, and a silicon-specific defect would go with it. So the row, the spec and the benchmark entry now all say what was measured WHERE: host arm 93/96 on sm_110 with fa2 and cutlass-fp8 both DISABLED, host arm 96/96 on sm_121a with both ENABLED, n=1 per arm. What is established is narrower and still worth having: on ONE box, holding the entire rest of the tower constant, flipping only the mamba arm moves 93/96 to 96/96. The `DenseBf16` reading stays the leading mechanism -- the golden's oracle is W8A8 and the host arm is W8A16 because `input_scale` is carried and not applied (nemotron_h.cpp:419-422) -- but it now carries the obligation to explain why GB10 is clean, and the honest reading is that the perturbation is marginal rather than grossly wrong. One proposed candidate is excluded rather than chased. "The two arms resolve different fp8 GEMMs" cannot be the differentiator, because that configuration runs NO fp8 GEMM on either box: `DenseFor` dequantizes and hands the result to vt::MatmulBT on the CPU queue (nemotron_h_device.cpp:2027). The device-side difference that IS checkable is attention -- CudaArchFeatures.cmake:349 provides fa2 for 12.1a and not for 11.0, so GB10 runs FlashAttention-2 over the 6 GQA layers where Thor runs the portable fallback. Next step named rather than the question closed: the oracle's top-2 margin at the three moved tokens, because this family has already produced a "divergence" that turned out to be a bit-exact near-tie; then a NemotronHTrace layer bisect; then a repeat to lift n=1. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
Correction to #1290: the divergence is an sm_110 result, and GB10 is a counter-measurementThe issue as filed said the host mamba arm is what
Left as written, the issue would have been dismissed the first time somebody checked on GB10 and got a pass — taking a real silicon-specific defect with it. #1290, its What is established is narrower and still worth having: on one box, holding the entire rest of the tower constant, flipping only the mamba arm moves One candidate excluded rather than chased"The two arms resolve different fp8 GEMMs" was the natural next hypothesis. It cannot be the differentiator, because the The device-side difference that is checkable is attention: Next step, named rather than the question closed
Preflight over the merged tree: **All gates green, |
…of its four behaviours ARE refusals The helper exists because `examples/nemotron_h_gen` reports neither a rate nor a duration, so the per-output-token number this row is measured on has to be derived. A derived number that prints 0, or a negative, or a rate over an unknown denominator is indistinguishable from a measurement once it reaches a report, so most of the helper is guards -- and nothing in the tree pinned them. Six cases now do: the rate is printed with the window, the excluded load and the token count beside it; the load is excluded rather than subtracted a second time; a non-positive window refuses; a missing TOKEN MATCH line refuses; zero compared tokens refuses; and the vLLM denominator is the pinned 0.014369 s rather than a constant that could drift and silently restate the gap. Armed, not merely present. Deleting the non-positive-window guard reds the suite; making the missing-token path print `per output token 0.000000 s` reds it. The tree was restored byte-for-byte between mutations and after, sha256 be32e5f5c786ccb283c88b88e256f9c0e5d79af87eeef5615a3c10a615d3ce8c both times, and the suite is green at that sha. The whole `run_gate` window change was also driven end to end against a fake driver and a fake nvidia-smi before it is allowed near the scarce GB10 lease: a 3 s load plus a 4 s decode yields 39 samples over a 3.994 s window rather than ~70 over ~7 s, so the load really is outside the brackets; a driver that dies during load reports `NO busy fraction is sampled` and still surfaces its exit code 3; and a 93/96 divergence surfaces exit 1 with `matched 93` beside the rate. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
The sampler fix is validated end-to-end before it touches the GB10 leaseA broken instrument fails toward a verdict about the code, so
The second row is the one that mattered: an instrument that swallowed the exit code while reporting a plausible fraction is exactly how an infrastructure failure comes to read as a statement about the model. The refusals are now pinned, and armed
Armed rather than present: deleting the non-positive-window guard reds the suite, and making the missing-token path print In flight
The 6.31% number is GB10's, but the hypothesis — that this arm raises the decode busy fraction — is testable by ON/OFF on any box, and Thor can answer it now. |
…se doctest stringifies char* as bool
The Thor re-run confirmed the repaired carry gate -- 5 cases, 63 assertions, 0
failed -- and printed the numbers that justify it: a noise floor of 0.2465 at
T=1, a carried conv window agreeing to 0.1746 against a band of 0.5, and a
carried SSM state agreeing to 0.0614 against the same band, at 2.9x and 8.1x
margins.
It also proved the diagnosis that drove the repair. A dropped carry separates the
second leg's output by 0.2045 while the noise the comparison must accept is
0.2465, so the defect really is smaller than the noise, `separation > 2 *
noise_floor` is false, and the case correctly asserts nothing there.
Correctly, but not legibly. The line that reports WHY no assertion was made
printed `1`. doctest stringifies a `const char*` as a bool, and the message
streamed a `char*` ternary, so it printed the truthiness of a non-null pointer
whichever branch was live -- an unconditional `1` that says nothing about the
condition it names.
That line exists to make "no assertion was made here" a STATED result rather than
a silent hole, so a version of it that cannot say what it means is the same class
of defect as the band it reports on: an instrument that formats like information.
It now builds a std::string. Reproduced against doctest 2.5.2 in both forms
first: `MESSAGE("x " << (false ? "yes" : "prose"))` prints `x 1`, and the
std::string form prints `x prose`.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
Thor re-run with the fixed sampler: the repaired carry gate holds, and its numbers justify the repair
The data now proves the diagnosis that drove the repair. A dropped carry separates the second leg's output by 0.2045, while the noise the comparison must accept is 0.2465 — the defect really is smaller than the noise, so One diagnostic was defective, and it is the kind that hidesThe line reporting why no assertion was made printed That line exists to make "no assertion was made here" a stated result rather than a silent hole, so a version that cannot say what it means is the same class of defect as the band it reports on: an instrument formatted like information. Reproduced against doctest 2.5.2 both ways before fixing — — and it now builds a Head is |
…ide a Thor decode The decode-window sampler works. With the arm on, Thor read 42.55% busy over 564 DECODE samples where the old conflated window read 15.33%, and the reason is now measured rather than argued: the engine load is 500.9 s and the decode is 75.4 s, so the old window was 87% load. The same run also printed `per output token 0.785606 s (vLLM 0.014369 s; ratio 54.7x)`. That ratio is wrong in the way the busy fraction was wrong an hour ago: 0.014369 s is a GB10 figure and this decode happened on Thor, so the quotient compares two different pieces of silicon. Fixing the busy-fraction surface and leaving its twin is how a bad comparison survives a correction, and 54.7x is exactly the kind of number that gets quoted once it exists. The rate still prints, because it is a real measurement of that box. Only the comparison is withheld, and it is withheld BY NAME rather than silently dropped, so the reader knows a ratio was declined rather than forgotten. The gate script now passes ARCH through. Two cases pin both arms, and the guard is armed: quoting the ratio unconditionally reds the suite. Tree restored byte-for-byte after the mutation, sha256 3d94497009c384bcb36709a48b3e1c5da203fea23db020ecb097b79af035cae1, green at that sha. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…n Thor, and the divergence reproduced The acceptance test of this unit was never a ratio: the GPU busy fraction had to rise, measured with its denominator. On the corrected instrument it does. Second Thor lease, fresh build and clone, same binary, same checkpoint, same golden, differing only by VT_NEMOTRON_H_DEVICE_MAMBA: arm ON 96/96 STRICT PASS rc=0 240/564 = 42.55% busy 0.785606 s/token arm OFF 93/96 DIVERGENCE rc=1 700/3808 = 18.38% busy 5.633442 s/token +24.17 points, a 2.31x rise in occupancy, and 7.17x less time per decode token. Peak host 44070 MiB. Zero reference-tier lines on either arm, so neither ran on the portable tier. READ ON THE BOX IT WAS TAKEN ON. These are sm_110 figures, and BOTH references this row is normally quoted against -- the 6.31% baseline and the 0.014369 s per token -- are GB10's. Neither supports a ratio against these numbers, the instrument withholds both by name off 121a, and the stale `ratio 54.7x` and `ratio 392.1x` strings in that run's log predate the per-token arch gate and must not be quoted. The per-token VALUES are sound; the ratios beside them are not. The divergence also reproduced, which lifts the n=1 caveat for Thor: two independent leases with separate builds agree exactly, 96/96 on and 93/96 off both times. GB10 stays n=1 in the other direction, and no GB10 run of the DEVICE arm exists at all yet -- so the hypothesis is SUPPORTED on sm_110 and UNMEASURED on sm_121a, and the owed list now says so rather than implying one covers the other. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
`origin/main` moved again while the second Thor lease was running. Merged rather than rebased, and the gate is re-run over the merged tree: AGENTS.md R4 says to merge an immutable SHA and re-run the full gate rather than reading the diff. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
The acceptance test is MET on sm_110: the decode busy fraction ROSE 18.38% to 42.55%Second Thor lease, fresh build and clone, sampler measuring the decode window alone. Same binary, checkpoint and golden; only
+24.17 points, a 2.31x rise in occupancy, and 7.17x less time per decode token. Peak host 44070 MiB. This is the measurement the unit was sent to produce, and it is the one the first run could not make: that run reported 15.33% vs 14.73% because its window was 87% engine load (500.9 s load against a 75.4 s decode). Fixing the window did not flatter the result — it revealed one. Read on the box it was taken onThese are sm_110 figures. Both references this row is normally quoted against — the 6.31% baseline and the 0.014369 s per token — are GB10's, so neither supports a ratio against these numbers. The instrument now withholds both by name off So: the occupancy hypothesis is SUPPORTED on sm_110 and UNMEASURED on sm_121a. The owed list says exactly that rather than letting one stand in for the other. The divergence reproduced, lifting n=1 on ThorTwo independent leases, separate builds and clones, agree exactly: Still owedThe GB10 run (the only one readable against either baseline, and the only place a device-arm A3 does not yet exist); the §5.1 per-block numeric gate against
|
`main` advanced to f06b9e9 carrying #1297, which routes `test_dspark_block_size_guard.cpp` through `tests/support/process_id.h`. This branch's `build-newest-gcc` failure was that inherited red, so it is merged forward to pick the fix up rather than re-run against a stale base. `docs/BENCHMARKS.md` was a GENUINE content conflict, not a driver artifact: this branch inserts the A2-Q1 row and main inserted the `ENG-RECORD-ANCHOR-RATCHET` row at the same point. They are different keys, so BOTH are kept -- main's first, then this row's. Neither was dropped, and no ratchet was raised to fit them. The union auto-merge of `.agents/issue-index.md` was DISCARDED: main's file taken wholesale, only rows whose KEY main lacks re-appended, main's file asserted a strict PREFIX. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
GitHub does not run the `merge=union` driver `.gitattributes:7` sets, so a branch touching `.agents/issue-index.md` re-conflicts whenever main appends a row. Resolved locally where the driver runs: the union auto-merge was DISCARDED, main's file taken wholesale, only rows whose KEY main lacks re-appended, and main's file asserted a strict PREFIX with zero duplicate keys. Any keyed-table collision kept BOTH sides, main's first, so no row is dropped. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
GitHub does not run the `merge=union` driver `.gitattributes:7` sets, so a branch touching `.agents/issue-index.md` re-conflicts whenever main appends a row. Resolved locally where the driver runs: the union auto-merge was DISCARDED, main's file taken wholesale, only rows whose KEY main lacks re-appended, and main's file asserted a strict PREFIX with zero duplicate keys. Any keyed-table collision kept BOTH sides, main's first, so no row is dropped. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
GitHub does not run the `merge=union` driver `.gitattributes:7` sets, so a branch touching `.agents/issue-index.md` re-conflicts whenever main appends a row. Resolved locally where the driver runs: the union auto-merge was DISCARDED, main's file taken wholesale, only rows whose KEY main lacks re-appended, and main's file asserted a strict PREFIX with zero duplicate keys. Any keyed-table collision kept BOTH sides, main's first, so no row is dropped. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…gates This branch is based on PR #1289 (`row/A2-Q1-fp8-mamba`), which is where `NemotronHMamba2MixerDevice` lives, so it did not start from `main`. The preflight's trailer and style gates SKIP when `origin/main` is not an ancestor of HEAD, and a skipped gate has reported nothing about this tree. `docs/BENCHMARKS.md` was a GENUINE content conflict, not a driver artifact: main updated the `ENG-RECORD-ANCHOR-RATCHET` row while this line carries the A2-Q1 row inserted at the same point. BOTH are kept, main's version of its own row first, then this line's. Neither was dropped and no ratchet was raised. `.agents/issue-index.md` auto-merged and was verified rather than trusted: main's file is a strict PREFIX of the result, with only rows whose KEY main lacks appended after it. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
2 participants
feat(#810 A2-Q1): NemotronH's 23 Mamba2 blocks reach the device on the FP8 W8A8 seam (#517)
The 23 Mamba2 layers were the last host bounce of a decode step, and they were
the expensive one. The host arm reaches its two projections through
Linear(..., const NemotronHOwned&), whoseDenseForcallsNemotronHOwned::DenseBf16()-- a FULL dequant of the fp8 tower into a freshbf16 buffer ON EVERY CALL. That is 23 x (10304x2688 + 2688x4096) = 890e6
elements re-expanded per token, plus a download of the normed hidden and an
upload of the mixer output per layer.
The block moves whole or not at all:
mixer.in_projproduces the fusedzxbcdtthat the causal conv and the SSD scan both consume (
mamba_mixer2.py:550, split:692-696), so there is no intermediate landing in which the conv is on thedevice and
in_projis not. That is why the shared FP8 W8A8 linear seam had tobe extracted first (#940,
dense_fp8_gemm.h), and this change routes through itrather than re-typing its entry points.
What it does
NemotronHMamba2MixerDevicemirrors the host arm statement for statement --same vt:: ops, same order, same dtypes, different backend:
The one substitution is the split: the host arm copies columns with
SliceColsbecause
vt::Mamba2ChunkScanvalidates every operand contiguous, andvt::QkvSplitis exactly that copy on the device.Both forwards select it at runtime, never by preprocessor.
MambaIsFp8namesonly
NemotronHWeightFormandvt::OpRegisteredis the op table's own answer,so a dense NemotronH or a device without the fp8 pair keeps the host bounce.
The paged arm hands the mixer the recurrent rows
vt::GdnStateGatheralreadygathered and zeroed and it advances them IN PLACE, so the two downloads, the
host mixer and the two uploads per layer are gone; A2-P's gather/scatter pair
and its
has_initial = truereasoning are unchanged.The
ssm_dtype == f32term in the paged selection is not decoration. The gatherwidens the page into an f32 working buffer by op contract and the host arm then
narrows it back to
ssm_dtype, so on a checkpoint whosemamba_ssm_cache_dtypeis not f32 the two arms would round differently and the per-block numeric gate
would compare two different computations.
Three points where this differs from the spec, deliberately
Fp8Weightpair is built ON FIRST DEVICE USE in aResidentSlottheweights own, not by the loader as spec §4.1/§4.2 proposed. That is A2-Q2a's
newer idiom and it is better here: it does not double the 890 MB tower in
host memory at load, it uploads nothing on a host-only run, and it leaves
rep.host_bytes(the literal18888922112pinned attest_nemotron_h_loader.cpp:310) untouched, so §4.2's re-derivationobligation does not arise. Keying on the slot rather than an address is
CUDA BF16 MoE resident cache reuses stale pointers after engine teardown #237: across two engine builds in one process an address-keyed cache returns
the previous engine's device pointers, which is plausible wrong values.
dense_fp8::ResidentFp8still does not callload_stats::AddDeviceUpload--that is FP8 W8A8 weights are uploaded without load accounting or the post-upload residency step, unlike every other resident weight in the same file #974, unchanged, and the shared header is not touched -- so A2-Q1
accounts what IT uploads at the site that causes it, as
ResidentWeightandResidentNvfp4do.ResidentFp8has uploaded it,so the conversion peaks at one projection rather than at the whole tower.
The comparison is not bit-comparable, and that is the point
The host reference is W8A16:
DenseBf16says outright thatinput_scaleiscarried and not applied. The device arm is W8A8, which is what vLLM does. The
difference between them is the e4m3 activation quantization, so every band in
tests/vllm/models/test_nemotron_h_mamba_device.cppis measured in the runagainst a defect the fixture separates, and every element count is asserted
against the geometry -- a maximum over zero elements is also 0.0.
The new file is the cheap arm in front of the real-checkpoint gate, for the same
reason A2-Q2a's device MoE file exists:
BuildTinyis all-kDense, so nothingin the existing device suite could reach the FP8 W8A8 arm at all.
G-SAFE
Untouched. All three clauses of the interlock at
nemotron_h_registry.cpp:162stand. This change creates no paging, no carriedstate and no batching that A2-P did not already create, and
num_reqs <= 1remains A2-B's to remove.
What is NOT proved here
The §5.1 per-block numeric gate on the real 20.1 GiB checkpoint, the A3
96/96 mode=decode STRICT PASSre-run, the §5.3 mutation pass and the GPU busyfraction are the unit's acceptance test and they need a GB10 lease.
scripts/nemotron-h-a2q1-dgx-gate.shrecords the recipe, including the threeenvironment facts that have each voided a run before: the CUDA lane is
sbsa,nvcc --versionis not a sufficient postcondition, and/workspaceis CIFS andrefuses symlinks. The spec's
## Nowand## Owedsay so, anddocs/BENCHMARKS.mdrecords the measurement as PENDING rather than as silence.Issue: #810 (parent row #517). Spec:
.agents/specs/nemotron-h-a2q1-fp8-mamba.md.FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]