What
On Jetson Thor (sm_110), tests/vt/test_ops_moe_grouped.cpp:1144 fails:
NVFP4 block8-vs-block16 M=8 K=4096 N=4096 bitdiff=15/32768
Two NVFP4 groupings that should agree disagree in 15 of 32768 output elements.
Why this one is different from the rest of the sm_110 baseline
#955 records 14 failures at 5a0ffe9e3, and thirteen of them are feature
absence: no FP8 op kernel, no vendored FA-2 on MLA prefill, two tests
hardcoding GB10. Those cannot be made green on this architecture and are not
defects in anything.
This one is not absence. marlin-nvfp4 is ENABLED for [110] — verified
from the arch gating, alongside fa2, cutlass-nvfp4{,-sm100}, cutlass-fp8,
scaledmm-c3x-* and fp4-mma, which are all disabled for that arch. So the
kernel under test is present and running, and it is disagreeing with itself
across two block sizes. That is a correctness signal on a live path.
Filed separately because #955 is a baseline issue, and a baseline issue is the
kind of record that gets closed wholesale as "known reds on this host". This
finding should not go with it. #955 states the reasoning in its text but not as
its own defect; it should point here instead.
Not established
- Whether GB10 (sm_121) shows the same disagreement. If it does, this is not
sm_110-specific and the title is wrong.
- Which of the two groupings is correct, or whether both are wrong.
- Whether 15/32768 is a rounding-boundary artifact or a genuine indexing or
accumulation defect. Bit-count alone does not distinguish those, and a small
bitdiff is not evidence of a small problem.
Reproduce with the container recipe in .agents/environment.md (Thor section,
landing in #956); the test is test_ops_moe_grouped.
Found during #699 W0.5, the Thor bring-up.
What
On Jetson Thor (sm_110),
tests/vt/test_ops_moe_grouped.cpp:1144fails:Two NVFP4 groupings that should agree disagree in 15 of 32768 output elements.
Why this one is different from the rest of the sm_110 baseline
#955 records 14 failures at
5a0ffe9e3, and thirteen of them are featureabsence: no FP8 op kernel, no vendored FA-2 on MLA prefill, two tests
hardcoding GB10. Those cannot be made green on this architecture and are not
defects in anything.
This one is not absence.
marlin-nvfp4is ENABLED for[110]— verifiedfrom the arch gating, alongside
fa2,cutlass-nvfp4{,-sm100},cutlass-fp8,scaledmm-c3x-*andfp4-mma, which are all disabled for that arch. So thekernel under test is present and running, and it is disagreeing with itself
across two block sizes. That is a correctness signal on a live path.
Filed separately because #955 is a baseline issue, and a baseline issue is the
kind of record that gets closed wholesale as "known reds on this host". This
finding should not go with it. #955 states the reasoning in its text but not as
its own defect; it should point here instead.
Not established
sm_110-specific and the title is wrong.
accumulation defect. Bit-count alone does not distinguish those, and a small
bitdiff is not evidence of a small problem.
Reproduce with the container recipe in
.agents/environment.md(Thor section,landing in #956); the test is
test_ops_moe_grouped.Found during #699 W0.5, the Thor bring-up.