docs(MODEL-MM-dots3-note): W0.5 — Thor is a CUDA host now, and its baseline caught a segfault on main (#699, #955, #960)

[localai-bot]

W0.5 of the dots3-note row (#699), the first dispatchable task under §6.4 option B. Docs and records only — no src/, no include/, no tests/. Issues: #699 (the row) and #955 (the sm_110 baseline this opens).

What was done on Thor

192.168.68.23 now builds vllm.cpp with CUDA ON for sm_110, runs kernels on the device, and has a recorded ctest baseline. The recipe is in .agents/environment.md so the next agent does not repeat the archaeology.

Container, not a host toolchain, and the reason is measured. / on Thor is a read-only 4.4 G ext2 loop (/dev/loop0, 1.3 G free) — it is an immutable Kairos image, so apt install into / is not available at all. A /home-prefix CUDA runfile would work but has to be re-derived after every reimage; the container carries the toolchain in one pinned digest and the NVIDIA runtime injects the host driver.

The image is a 4-line Dockerfile on nvidia/cuda@sha256:7d2f6a8c2071… (pinned by digest — the 13.0.1 tag moves) plus cmake ninja-build git python3 python3-dev ca-certificates shellcheck. Inside: nvcc 13.0.88, cmake 3.28.3, ninja 1.11.1. Every invocation needs --runtime=nvidia, -e NVIDIA_DISABLE_REQUIRE=1 and sudo -n.

That last point corrects a script already on the box: /home/mudler/_build_thor.sh states the container runtime "REFUSES this image on its driver" and drops --runtime on that basis. It is wrong, and it is wrong only because it also drops NVIDIA_DISABLE_REQUIRE=1.

Proof the build is really a CUDA build

A build that quietly disables CUDA is the failure mode this task exists to catch, so it is checked three independent ways:

• configure prints CUDA target architectures: 110 and resolves /usr/local/cuda/bin/nvcc (NVIDIA 13.0.88);
• 30 *.cu.o objects, each containing exactly one cubin, all sm_110 — cuobjdump --list-elf over every object reads 30 sm_110. 30 of the tree's 53 .cu files is correct, not partial: fa2, cutlass-nvfp4, cutlass-nvfp4-sm100, cutlass-fp8, scaledmm-c3x-sm90, scaledmm-c3x-sm100 and fp4-mma all resolve DISABLED for [110], only marlin-nvfp4 is ENABLED, and configure says the TU set is narrowed by VT_CUDA_FEATURE_TABLE. CUTLASS is absent and supplying it changes nothing — the cells are arch-gated, not CUTLASS-gated;
• ldd libvllm.so resolves libcudart.so.13 and libcublasLt.so.13.

Runtime, on the device: test_cuda_backend reports CUDA compute capability: sm_110 and pageable=1 integrated=1 UnifiedMemory=true, 6/6 cases and 25/25 assertions. A hand-written nvcc -arch=sm_110 kernel also launches and returns correct values.

Build was -j4 on purpose (vm.overcommit_memory=1, zero swap, no OOM killer — the box reboots instead). It cost nothing: 1460/1460 targets in ~20 min at peak ~6 GB of 122 GB.

The baseline — 14 red, and that is the deliverable

ctest -j4 --timeout 1800 at 5a0ffe9e3: 484 tests, 468 passed / 2 skipped / 14 FAILED, 177.89 s. All 14 reproduce at -j1, so none is a starvation artefact. Nothing was fixed; the full table with per-test first-failing assertions is in .agents/environment.md and in #955.

They collapse into four causes. Ten are the build honestly refusing what sm_110 lacks — five throw vt: no kernel for op QuantFp8Static (id 52), four throw built without the vendored FlashAttention-2 on MLA prefill. Two hardcode GB10: test_platform:307 asserts is_device_capability_family(120), and test_op_parity:2487 replays a dgx-captured output_cbor_sha256 inside a case that names itself dgx-only and runs anyway. test_capi's SIGSEGV and test_linear_method's un-run MXFP4 fused path are already red on GB10 (#907).

The one substantive finding is test_ops_moe_grouped:1144: NVFP4 block8-vs-block16 M=8 K=4096 N=4096 bitdiff=15/32768. marlin-nvfp4 IS enabled for [110], so unlike the other groups this is a live kernel disagreeing with itself across block sizes on hardware that runs it. Possibly adjacent to #325 (same cell, same device) but that is a throughput claim and this is a correctness one.

Two things stated plainly rather than worked around

The spec's W0.5 gate is not met. It read "the existing suite passes there", and it cannot: ten of the 14 reds are feature absence this row cannot make green. The spec now says so and replaces it with the gate that actually binds — a row regresses on Thor only if it lengthens the list of 14.

One false red was mine, briefly. test_serve_low_tools failed the first run with FileNotFoundError: 'shellcheck': tests/tools/test_online_gate_startup.py:260 shells out and raises rather than skipping when the binary is absent — an absent instrument reading as a code verdict. That was a property of my image, not of Thor. shellcheck is in the recorded Dockerfile and the test passes; the 14 above are with it installed.

Also corrected in the records

nvidia-smi on Thor was never broken — it works under sudo -n and fails only unprivileged, so NvRmMemInitNvmap failed: Permission denied was a privilege problem. Its memory columns read [N/A] because this is an integrated GPU with no VRAM counters, so read free -g instead. ~/gpu.lock exists again (the 2026-08-11 note saying it does not is stale) and a host local-ai worker is back.

Verification

scripts/agent-preflight.sh --staged is green on every record gate that applies (check-agent-record, doc-checkpoint --staged, now-current --staged, trailer and commit-style suites). The other failures it reports — check-release-binary-contract, check-release-workflow, check-test-registration and their three mutation suites — reproduce identically on a clean main and are unrelated to this diff, and test_cpu_x86_llamacpp_floor is #618 firing at loadavg 195 on the dev box.

No lifecycle state changed, so no STATUS/BENCHMARKS write is owed; the row stays SPIKE with no engine code. .agents/issue-index.md is appended at the end only.

🤖 Generated with Claude Code

[localai-bot]

Rebased onto 2daa3287f (PR #934) and re-measured. New head b1ae13538.

The rebase pulled 37 files across src/, tests/ and CMakeLists.txt (+4093/-118) into the tree the baseline measured, including src/vt/cuda/cuda_matmul_fp8_cutlass.cu and a new tests/vt/test_ops_fp8_cpu.cpp, so the baseline was re-run rather than carried over. It moved, and the PR is retitled because of what it found.

	5a0ffe9e3	2daa3287f
total	484	485
failed	14	15
FP8 group	Failed — clean throw vt: no kernel for op QuantFp8Static (id 52) on device cuda	SEGFAULT

The FP8 group did not just get worse, it changed character: the loud refusal became [vt reference-tier] op=QuantFp8Static device=cuda has NO native kernel; running the PORTABLE CPU fallback (correct but slow) followed by SIGSEGV. Five clean refusals became five crashes, and #842's new test_ops_fp8_cpu adds a sixth, crashing at :279 in the case named "G2: CPU QuantFp8Static equals CUDA QuantFp8Static byte for byte" — the test name states the mechanism. cutlass-fp8 is ENABLED on GB10 and DISABLED for [110], so on the gate host the native kernel exists, the fallback is unreachable, and no CI lane could see it.

Filed as #960, not fixed here. Attribution to f270b4b0a (#842) is indicated by the diff and by the reference-tier log line; it is not bisected, and the issue says so.

Records updated to the new base throughout: environment.md's table (new test numbers 382/385/390/415/416/418/433), the ## Now and §7 W0.5 entries in the spec, and two issue-index.md rows appended at the end (#955 restated at the new SHA, #960 added). .agents/issue-index.md diffs as exactly +2 lines against origin/main with nothing removed.

Spec reconciled by section, not by conflict marker. .agents/specs/dots3-note.md was taken from origin/main wholesale and my three scoped edits reapplied on top, so §6.4's option-B decision, the new ## Owed section and 934's framing survive untouched. ## Now is rewritten so there is ONE account of the state: it reports W0.5 landed, points at the environment.md recipe rather than restating the Thor facts, and names W1 as next. One further staleness left by #934 is fixed in passing — §7's lead-in still read "Nothing past it is dispatched until §6.4 is answered", which is no longer true.

The differential gate reasoning now lives in the spec, not only here (§7's W0.5 entry): all-green is the wrong bar on a host whose arch legitimately lacks features, because it would either block every brick forever or invite someone to weaken a test; a row regresses on Thor only if it lengthens the list. The spec also now says to re-measure whenever the base moves across src/, tests/ or CMakeLists.txt — this rebase is the evidence for why.

scripts/agent-preflight.sh --staged reports All gates green at the new head; the release and test-registration gates that were red at the previous base are green on current main. Still not merged.