diff --git a/.agents/environment.md b/.agents/environment.md index 39104eafc..b420350db 100644 --- a/.agents/environment.md +++ b/.agents/environment.md @@ -179,6 +179,182 @@ environment: - `nsys` from `nsight-systems-cli` in that image is **2024.2.3 and cannot trace CUDA here** ("does not contain CUDA trace data"). Do not plan a graph/kernel-count measurement on Thor without first installing a newer one. + - **★ PROVISIONED AS A BUILD-AND-TEST HOST, 2026-08-15** + (`dots3-note` W0.5, issue [#699](https://github.com/mudler/vllm.cpp/issues/699)). + The recipe below was executed end to end that day: toolchain, CUDA-ON build + for sm_110, a kernel that ran on the device, and a full `ctest` baseline. + Copy it rather than re-deriving it. + + **Why a container and not a host toolchain.** `/` is a **read-only** 4.4 G + `ext2` loop (`/dev/loop0`, 1.3 G free) — `kairos-4db2` is an immutable + Kairos image, so `apt install` into `/` is not available at all. `/home` is + the only writable volume (918 G, ~362 G free). A `/home`-prefix CUDA + runfile install would work but would have to be re-derived after every + reimage; the container carries the whole toolchain in one pinned digest and + the NVIDIA container runtime injects the host driver, so it survives + reimages. + + **1. Toolchain image.** Build it once from a 4-line Dockerfile pinned to the + base image *by digest* (not by tag — the `13.0.1` tag moves): + + ```dockerfile + FROM nvidia/cuda@sha256:7d2f6a8c2071d911524f95061a0db363e24d27aa51ec831fcccf9e76eb72bc92 + ENV DEBIAN_FRONTEND=noninteractive + RUN apt-get update -qq \ + && apt-get install -y -qq --no-install-recommends \ + cmake ninja-build git python3 python3-dev ca-certificates \ + && rm -rf /var/lib/apt/lists/* + ``` + + That digest is `nvidia/cuda:13.0.1-devel-ubuntu24.04` as resolved on Thor on + 2026-08-15. `sudo -n docker build -t vllmcpp-thor:cuda13.0.1 .` (docker + needs `sudo`; the user is in group `admin`, not `docker`). Inside: + **nvcc 13.0.88, cmake 3.28.3, ninja 1.11.1, python 3.12.3**. A live copy of + the Dockerfile and the three driver scripts sits in `/home/mudler/thor-w05/` + on the box; the repo is the authority if they disagree. + + **2. Every container invocation needs the same three flags:** + `--runtime=nvidia` (NOT `--gpus all`), `-e NVIDIA_DISABLE_REQUIRE=1`, and + `sudo -n`. The pre-existing `/home/mudler/_build_thor.sh` says the runtime + "REFUSES this image on its driver" and therefore omits `--runtime`; that is + **wrong, and it is wrong only because it also omits + `NVIDIA_DISABLE_REQUIRE=1`**. With the flag, `cudaGetDeviceCount` returns 1, + `cudaGetDeviceProperties` reports `NVIDIA Thor sm_110`, and a hand-written + `nvcc -arch=sm_110` kernel launches and returns correct values. + + **3. Source transfer.** `git archive --format=tar HEAD` from the dev box, + `scp`, untar into a FRESH directory under `/home` (never rsync — see + [[dgx-transfer-git-archive-not-rsync]]). Record the base SHA beside it. + + **4. Configure and build** (bind-mount the checkout, no GPU needed to + compile but the flags are harmless): + + ```sh + sudo -n docker run --rm --runtime=nvidia -e NVIDIA_DISABLE_REQUIRE=1 \ + -v "$SRC":/src -w /src vllmcpp-thor:cuda13.0.1 \ + cmake -S /src -B /src/build-cuda -G Ninja -DCMAKE_BUILD_TYPE=Release \ + -DVLLM_CPP_CUDA=ON -DVLLM_CPP_CUDA_ARCHITECTURES=110 -DVLLM_CPP_TRITON=OFF + sudo -n docker run --rm --runtime=nvidia -e NVIDIA_DISABLE_REQUIRE=1 \ + -e CMAKE_BUILD_PARALLEL_LEVEL=4 -v "$SRC":/src -w /src \ + vllmcpp-thor:cuda13.0.1 cmake --build /src/build-cuda -j 4 + ``` + + **`-j4` is deliberate** — see the overcommit warning below. 1460/1460 + targets in **~20 minutes** at `-j4`, peak host memory ~6 GB of 122 GB, so + the conservative setting costs almost nothing here. + + **5. Prove the build is really a CUDA build, because a silent CPU fallback + is the failure mode.** Three independent checks, all measured 2026-08-15: + - configure prints `CUDA target architectures: 110` and finds + `/usr/local/cuda/bin/nvcc` (NVIDIA 13.0.88); + - **30 `*.cu.o` objects** are compiled and every one of them contains + exactly one cubin, `sm_110` — + `for f in $(find build-cuda -name '*.cu.o'); do cuobjdump --list-elf $f; done | grep -o 'sm_[0-9]*' | sort | uniq -c` + reads `30 sm_110`. 30 of the tree's 53 `.cu` files is CORRECT, not a + partial build: configure says `feature TUs narrowed by + VT_CUDA_FEATURE_TABLE`, and on sm_110 `fa2`, `cutlass-nvfp4`, + `cutlass-nvfp4-sm100`, `cutlass-fp8`, `scaledmm-c3x-sm90`, + `scaledmm-c3x-sm100` and `fp4-mma` all resolve DISABLED. Only + `marlin-nvfp4` is ENABLED for `[110]`. CUTLASS is absent and supplying it + would change nothing — the cells are arch-gated, not CUTLASS-gated. + - `ldd build-cuda/libvllm.so` resolves `libcudart.so.13` and + `libcublasLt.so.13`. + + **6. Runtime proof, on the device.** `./tests/test_cuda_backend` under + `--runtime=nvidia` reports `CUDA compute capability: sm_110` and + `pageable=1 integrated=1 UnifiedMemory=true`, 6/6 cases and 25/25 + assertions. `test_cuda_ops` executes the op kernels themselves. + + **7. `nvidia-smi` on the host** works under `sudo -n` and only under it — + unprivileged it dies with `NvRmMemInitNvmap failed: error Permission + denied`, which is a PRIVILEGE problem and was previously mis-recorded as a + broken driver. Its memory columns read `[N/A]`: this is an integrated GPU + with no separate VRAM counters, the same shape as + [[gb10-has-no-dram-counters-ncu-memory-pct-is-a-lie]]. Do not build a + memory-headroom check on `nvidia-smi` here; read `free -g` instead. + + **`~/gpu.lock` EXISTS again** (the 2026-08-11 line above saying it does not + is stale). Wrap GPU work as + `flock /home/mudler/gpu.lock -c ''`. A host `./local-ai + worker` process is also back and was running throughout this baseline; it + was idle and did not perturb it, but stop it before any measurement. + + **8. `ctest` BASELINE — `2daa3287f`, 2026-08-15, `ctest -j4 --timeout 1800` + inside the image above, 103.35 s wall:** + + ```text + 485 tests: 468 passed | 2 skipped | 15 FAILED + ``` + + Skipped: `test_modelopt_mixed_precision_checkpoint` and `test_voxtral_e2e`, + both for an absent checkpoint. Reproduce with + `/home/mudler/thor-w05/ctest.sh`. These are the sm_110 baseline and are NOT + to be "fixed" by a row that merely builds here; **a row is a regression on + Thor only if it lengthens this list.** Recorded as + [#955](https://github.com/mudler/vllm.cpp/issues/955), the sm_110 + counterpart of [#907](https://github.com/mudler/vllm.cpp/issues/907). + + | # | Test | First failing assertion | Root cause | + |---|---|---|---| + | 80 | `test_platform` | `test_platform.cpp:307` `CHECK(cu.is_device_capability_family(120))` false | the TEST hardcodes the sm_12x family. Thor is 11.0 | + | 82 | `test_linear_method` | `:246` `after == before + 1` → `0 == 1`, case "MXFP4 fused gate_up … fused path ran" | the MXFP4 fused path does not run on sm_110. Also red on GB10 ([#907](https://github.com/mudler/vllm.cpp/issues/907)) | + | 86 | `test_qwen3_5_gdn_spec_routing` | **SEGFAULT** | FP8 fallback crash, [#960](https://github.com/mudler/vllm.cpp/issues/960) | + | 91 | `…_glue_fuse_off` | **SEGFAULT** | same | + | 92 | `…_fused_chain_off` | **SEGFAULT** | same | + | 124 | `test_deepseek_v2_forward` | `:559` THREW `cuda mla_prefill_attention: built without the vendored FlashAttention-2` | no FA-2 on sm_110 | + | 335 | `test_capi` | `test_capi.cpp:487` SIGSEGV | pre-existing, arch-independent; same crash on GB10 ([#907](https://github.com/mudler/vllm.cpp/issues/907)) | + | 381 | `test_ops_fp8_cutlass` | **SEGFAULT** | FP8 fallback crash, [#960](https://github.com/mudler/vllm.cpp/issues/960) | + | 382 | `test_ops_fp8_cpu` | `:279` SIGSEGV in "G2: CPU QuantFp8Static equals CUDA QuantFp8Static byte for byte" | same, and this is the test that names the mechanism | + | 385 | `test_ops_moe_grouped` | `:1144` `bitdiff == 0` → `15`, logged `NVFP4 block8-vs-block16 M=8 K=4096 N=4096 bitdiff=15/32768` | **the one substantive standing sm_110 finding.** `marlin-nvfp4` IS enabled for `[110]`, so this is a live kernel disagreeing with itself across block sizes, not an absent feature | + | 390 | `test_ops_fused_chain` | **SEGFAULT** | FP8 fallback crash, [#960](https://github.com/mudler/vllm.cpp/issues/960) | + | 415 | `test_ops_mla_prefill` | `:340`, `:437` FA-2 absent | no FA-2 on sm_110 | + | 416 | `test_ops_mla_chunked_context` | `:790` FA-2 absent | same | + | 418 | `test_mla_attention_block` | `:999`, `:1044` FA-2 absent | same | + | 433 | `test_op_parity` | `:2487` `output_cbor_sha256` mismatch in "qwen27 GDN BA BF16 projection matches vLLM 0.25 oracle (**dgx-only**, CUDA)" | a dgx-captured golden replayed on Thor; the case names itself dgx-only and runs anyway | + + So the 15 collapse into four causes: **six** are + [#960](https://github.com/mudler/vllm.cpp/issues/960), **four** are the + missing vendored FA-2, **two** are tests that hardcode GB10 + (`test_platform`'s capability family, `test_op_parity`'s dgx-only golden), + and **three** stand alone — `test_capi`'s segfault and `test_linear_method`, + both already red on GB10, plus `test_ops_moe_grouped`. + + **The FIRST thing this lane found, and the argument for keeping it.** The + baseline was measured twice, at `5a0ffe9e3` and again at `2daa3287f` after a + rebase, and it MOVED. At the earlier SHA the FP8 group failed by throwing + `vt: no kernel for op QuantFp8Static (id 52) on device cuda` — a loud, + correct refusal, 484 tests / 14 red. At the later SHA the same request + silently takes the portable CPU reference tier and **segfaults**, 485 tests / + 15 red. `cutlass-fp8` is ENABLED on GB10 and DISABLED for `[110]`, so on the + GB10 host the native kernel exists, the fallback is unreachable, and nothing + in CI could see it. Full detail and the attribution in + [#960](https://github.com/mudler/vllm.cpp/issues/960). + + Two lessons for whoever runs this next. **Re-measure the baseline whenever + the base SHA moves across `src/`, `tests/` or `CMakeLists.txt`** — a stale + baseline is worse than none, because the next agent reads a regression as + the floor. And an earlier full run at `5a0ffe9e3` confirmed all 14 failures + of that generation reproduced serially at `-j1`, so `ctest -j4` is not + producing starvation artefacts on this box; re-confirm that only if a NEW + name appears. + + Two things NOT to conclude from this table. The FP8 and FA-2 groups are + **feature absence surfacing as a thrown exception**, which is the loud + failure the seam is supposed to produce — they are not sm_110 numerical + bugs. And the FA-2 message reads *"MLA prefill on **sm_121** IS + FlashAttention"* while running on sm_110: the text is hardcoded to the GB10 + arch, so do not read an arch out of it. + + **A trap this baseline walked into first.** `test_serve_low_tools` failed + the initial run with `FileNotFoundError: 'shellcheck'` — + `tests/tools/test_online_gate_startup.py:260` shells out to `shellcheck` and + raises rather than skipping when the binary is absent. That is an ABSENT + INSTRUMENT reading as a code verdict, and it was a property of the image, + not of Thor. `shellcheck` is in the Dockerfile above for exactly that reason + and the test passes; if you build the image without it, expect an extra + failure that means nothing. The harness defect is real and is NOT fixed by + that Dockerfile line — putting the binary in the image hides it here and + leaves it armed everywhere else. - **★ THIS BOX REBOOTS INSTEAD OF OOM-KILLING — size every load for it.** `vm.overcommit_memory=1` ("always overcommit") with **zero swap**: the kernel grants memory it cannot back, and touching those pages takes the WHOLE MACHINE diff --git a/.agents/issue-index.md b/.agents/issue-index.md index aad878c4b..27bc41f19 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -251,3 +251,5 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#874](https://github.com/mudler/vllm.cpp/issues/874) | — | `windows-msvc-cpu`/`windows-msvc-vulkan` still start on a CLOSED pull request: `check-release-workflow.py::validate_pr_ci` compares their whole job mapping for equality, so neither an `if:` clause nor a `needs:` guard can be added. Listed under `## Owed` in [`ci-concurrency.md`](specs/ci-concurrency.md) | bug | | [#949](https://github.com/mudler/vllm.cpp/issues/949) | — | Nothing in the tree refuses a borrowed `vt::Tensor` that outlives the object owning its storage, and the ONLY instrument that catches one is `sanitize-cpu`, which is `continue-on-error` — that is how [#904](https://github.com/mudler/vllm.cpp/issues/904) landed. Measured in the #936 review rather than argued: with the #904 fix reverted, a plain Release build with no sanitizer runs the case 18/18 passed, 546 assertions, `rc=0`, because `dtype` lives in the `vt::Tensor` struct and not in the freed buffer, so no ordinary gate can see the dangling read. Three remedies are open and none is foregone: promote the lane once it has a `main` baseline, add a test that fails without a sanitizer, or reject the pattern statically — a prototype detector for a member access chained onto a call returning an owning type by value swept 1777 files with no hit but the defect. Anchors: the owning deleter `src/vllm/model_executor/models/ltx2_device.cpp:1088 @ 800dd082f`, the read `src/vt/cpu/cpu_layernorm.cpp:33 @ 800dd082f`. Listed under `## Owed` in [`ltx2-device-staged-view-uaf.md`](specs/ltx2-device-staged-view-uaf.md) | bug | | [#933](https://github.com/mudler/vllm.cpp/issues/933) | `ENG-EXPERT-STREAM` | Measure gateability of the `llama-cpp-unsloth` oracle by BUILDING it and RUNNING `Qwen3.8-2.4T-A95B UD-Q1_0` on it. The oracle is pinned at `36fe8e1cc` (branch `iq1-narrow`) and records `gateable = no`, because the IQ1_XXXS port is grounded in the fork's SOURCE, read and cited, which is weaker than a running comparison. It is the only place ggml type 66 is defined: the vllm.cpp pin `237ad9b96` ends at `Q1_0 = 41` and `ggml-org` master `ad1de39e0` at `Q2_0 = 42`, while type 66 carries 96.92 % of that checkpoint's parameters. Running it needs the full 370 GiB checkpoint and, per Unsloth's documentation, at least 450 GB of RAM. Until then the ported arm has no running oracle, which is what `gateable = no` makes visible | task | +| [#955](https://github.com/mudler/vllm.cpp/issues/955) | `BACKEND-CUDA-SM110` | sm_110 (Jetson Thor) `ctest` baseline at `2daa3287f`: 468 passed / 2 skipped / **15 red** of 485. Four are the build honestly refusing what the arch lacks — `built without the vendored FlashAttention-2` on MLA prefill. Two hardcode GB10: `test_platform:307` asserts `is_device_capability_family(120)`, and `test_op_parity:2487` replays a dgx-captured `output_cbor_sha256` in a case that names itself dgx-only and runs anyway. `test_capi`s SIGSEGV and `test_linear_method`s un-run MXFP4 fused path are already red on GB10 ([#907](https://github.com/mudler/vllm.cpp/issues/907)). Six are the FP8 fallback crash split out as [#960](https://github.com/mudler/vllm.cpp/issues/960). The one substantive standing finding is `test_ops_moe_grouped:1144`, `NVFP4 block8-vs-block16 M=8 K=4096 N=4096 bitdiff=15/32768` — `marlin-nvfp4` IS enabled for `[110]`, so that is a live kernel disagreeing with itself, not an absent feature. Recipe and table in [`environment.md`](environment.md) | bug | +| [#960](https://github.com/mudler/vllm.cpp/issues/960) | `BACKEND-CUDA-SM110` | Asking for `QuantFp8Static` on a CUDA device with NO native FP8 kernel used to throw `vt: no kernel for op QuantFp8Static (id 52) on device cuda`; since [#842](https://github.com/mudler/vllm.cpp/issues/842) registered the static fp8 W8A8 pair on the CPU backend it silently takes the portable CPU reference tier instead and **SEGFAULTS**. Measured at `tests/vt/test_ops_fp8_cpu.cpp:279`, case "G2: CPU QuantFp8Static equals CUDA QuantFp8Static byte for byte", which takes the process down so G3 and the pair case never execute. Turned five clean sm_110 refusals into five crashes and added a sixth between `5a0ffe9e3` and `2daa3287f`. Unreachable on GB10 because `cutlass-fp8` is ENABLED there and DISABLED for `[110]`, so it needs a CUDA arch outside the cutlass-fp8 set to appear at all — the first finding of the newly provisioned sm_110 lane. Attribution indicated by the diff and the reference-tier log line, NOT bisected | bug | diff --git a/.agents/specs/dots3-note.md b/.agents/specs/dots3-note.md index 1b1b694bf..44dd4149a 100644 --- a/.agents/specs/dots3-note.md +++ b/.agents/specs/dots3-note.md @@ -389,11 +389,16 @@ Three facts follow, and they are recorded rather than worked around. 122 GB of RAM and its 123 GiB of free disk — the checkpoint will not even land. Designating the host does not change §6.2; it fixes *where our arm and our unit gates run*, which is a real and separate thing. -2. **Thor needs provisioning first** (W0.5): CUDA toolkit, cmake/ninja, the - `nvidia-smi` permission for the agent's ssh session, and — if any oracle work - is ever to run there — a vLLM build. Per - [environment.md](../environment.md) the box was reimaged 2026-08-11 and came - back with no host CUDA toolkit; this probe confirms that is still true. +2. **Thor needed provisioning first** (W0.5) — **DONE 2026-08-15**, recipe and + `ctest` baseline in [environment.md](../environment.md). Two corrections to + the read-only probe above. `nvidia-smi` was never broken: it works under + `sudo -n` and fails only unprivileged, so `NvRmMemInitNvmap failed: + Permission denied` was a privilege problem, not a driver one. And the + toolchain does not go on the host at all — `/` is a read-only loop on an + immutable image, so the CUDA toolkit lives in a digest-pinned container and + the box keeps no host CUDA. Free disk on `/home` measured 362 GB, not the + 123 GiB the earlier probe read. Still owed if oracle work is ever wanted + here: a vLLM build, which §6.2 says cannot serve THIS model regardless. 3. **Thor's standing traps apply.** `vm.overcommit_memory=1` with zero swap: the kernel grants memory it cannot back and touching those pages takes the whole machine down (observed three times on 2026-08-11). Any run here is sized @@ -445,13 +450,48 @@ Whichever is chosen, **no ceiling is declared** and the gap stays open. ## 7. Phases -W0 is this document. Nothing past it is dispatched until §6.4 is answered. +W0 is this document. §6.4 is answered — option B — so the phases below are +dispatchable in order, under the constraints that answer imposes. - **W0 — scope (this).** Arch map, reuse-vs-new, config traps, quant/HW fit, oracle plan, rows. **DONE.** -- **W0.5 — provision Thor.** CUDA toolkit, cmake/ninja, `nvidia-smi` under the - agent's ssh, a built vllm.cpp, and the ctest baseline green on sm_110. Gate: - the existing suite passes there before this row adds anything to it. +- **W0.5 — provision Thor. DONE 2026-08-15.** The recipe is + [environment.md](../environment.md), under the Jetson Thor profile: a + digest-pinned CUDA 13.0.1 container (nvcc 13.0.88, cmake 3.28.3, ninja 1.11.1) + run `--runtime=nvidia -e NVIDIA_DISABLE_REQUIRE=1` over a `git archive` + checkout on `/home`. A HOST toolchain was rejected on evidence, not taste: + `/` is a read-only 4.4 G loop on an immutable Kairos image, so `apt install` + into it does not exist. The build is CUDA-real and proved three ways — + configure prints `CUDA target architectures: 110`, all 30 `*.cu.o` carry + exactly one `sm_110` cubin each, and `libvllm.so` links `libcudart.so.13`. A + kernel ran on the device: `test_cuda_backend` reports `sm_110`, + `integrated=1`, `UnifiedMemory=true`, 25/25 assertions. + + **The gate as written — "the existing suite passes there" — is NOT met, and + it was the wrong gate.** At `2daa3287f` the baseline is 485 tests, **468 + passed / 2 skipped / 15 red**. Four are the build correctly refusing what + sm_110 does not have (no vendored FA-2), two are tests that hardcode GB10, two + are already red on GB10, and six are one FP8 defect + ([#960](https://github.com/mudler/vllm.cpp/issues/960)). None of those can be + made green by this row and none is this row's debt. Asking for "all green" on + a host whose arch legitimately lacks features would either block every brick + forever or invite someone to weaken a test to pass. **The gate that actually + binds is therefore differential: a row regresses on Thor only if it lengthens + that list.** The list itself, with per-test first-failing assertions, is + [#955](https://github.com/mudler/vllm.cpp/issues/955), the sm_110 counterpart + of [#907](https://github.com/mudler/vllm.cpp/issues/907). + + That gate has to be re-derived, not remembered: the baseline was measured at + two SHAs a few hours apart and it MOVED (484/14 → 485/15), because a change on + `main` turned a clean FP8 refusal into a segfault. **Re-measure whenever the + base moves across `src/`, `tests/` or `CMakeLists.txt`.** + + Two consequences this row carries forward. Thor's MLA prefill throws rather + than computes, so the W3/W4 attention bricks cannot be verified end to end + here on the FA-2 path at all — their gate stays the in-test double-precision + reference of §5, exactly as §6.4 requires under option B. And W9's + blockwise-FP8 arm has no native kernel on this box, and the fallback that + stands in for it currently crashes, so that arm is owed rather than pending. - **W1 — config + registry.** `dots3_note` in `hf_config.cpp` with RED-first assertions on all six §4 traps; `dots3_note_registry.cpp` as an additive TU registering `Dots3NoteForCausalLM` (and `Dots3NoteMTPModel` as INVENTORIED). @@ -541,20 +581,26 @@ Carried openly under option B (§6.4), not waived: ## Now W0 complete; **§6.4 answered on 2026-08-15 with option B**, so the row is no -longer blocked on a decision. It stays `SPIKE` until W1 lands code. - -In flight: **W0.5 — provision Thor at `192.168.68.23`**. Measured 2026-08-15, -and it shapes the work: Thor runs a Kairos-style immutable OS with `/` mounted -READ-ONLY (4.4G loop, 1.3G free), so `apt install` is unavailable; but Docker is -present, `/home` has 362G, the driver is at -`/opt/nvidia/l4t-gpu-libs/nvgpu/libcuda.so*`, and `nvidia-smi` under `sudo` -reports `NVIDIA Thor, 11.0` — compute capability **11.0 = sm_110**. The earlier -"NvRmMemInitNvmap failed: Permission denied" was a privilege problem, not a -broken driver. There is no CUDA toolkit on the box at all. - -W0.5 is not dots3-specific: Thor is the project's only non-GB10 CUDA host, so -its recipe and `ctest` baseline belong in `.agents/environment.md` and unblock -any row that wants sm_110 coverage. - -After W0.5, W1 (config + registry, with the six §4 traps RED-first) is the first -dots3 brick. +longer blocked on a decision. **W0.5 landed the same day.** The row stays +`SPIKE` until W1 lands code — provisioning a host is not porting a model. + +**W0.5 — DONE.** Thor at `192.168.68.23` builds vllm.cpp with CUDA ON for +sm_110 in a digest-pinned CUDA 13.0.1 container, runs kernels on the device, and +has a recorded `ctest` baseline of 468 passed / 2 skipped / **15 red** of 485 at +`2daa3287f`. The full recipe — image digest, the three container flags, the +build command, why `nvidia-smi` needs `sudo -n`, and the per-test failure table +— is in [environment.md](../environment.md) under the Jetson Thor profile, not +here: Thor is the project's only non-GB10 CUDA host, so it belongs to every row +that wants sm_110 coverage rather than to this one. The earlier probe's +"`nvidia-smi` refuses" reading was a privilege problem, and the toolchain lives +in a container because `/` is a read-only loop on an immutable image. + +The W0.5 gate as originally written ("the existing suite passes there") is not +met and was the wrong gate; §7 records the differential gate that replaces it +and the reasoning for it. The lane earned its keep immediately by finding +[#960](https://github.com/mudler/vllm.cpp/issues/960) — an FP8 refusal on `main` +that became a silent portable-CPU fallback and a segfault, invisible on GB10. + +**Next dispatchable: W1 — `dots3_note` config + registry**, with the six §4 +traps RED-first, and owing the §8.1 heading restructure in the same change as +the lifecycle move to `ACTIVE`.