ENG-EXPERT-STREAM: decode the two encodings the Qwen3.8-2.4T checkpoints are 97% made of

[localai-bot]

ENG-EXPERT-STREAM streams routed expert slices off NVMe, and never asked whether this tree can DECODE the bytes it streams. It cannot, or could not: each published Qwen3.8-2.4T-A95B checkpoint was missing exactly one encoding, and that encoding is 96.92 % of the model. A slice the streamer can address but not decode moves bytes for nothing, so this is the row's precondition rather than an adjacent nicety.

Closes nothing on its own; it is W9 and W12 of .agents/specs/expert-streaming.md, tracked by #912. Files #933 for what the new oracle still owes.

What the measurement found

Every published GGUF of unsloth/Qwen3.8-2.4T-A95B-GGUF at revision 567d3e6ac2 was censused by parsing tensor headers over HTTP range requests, no full download. Coverage is total rather than sampled: 1702 tensor records against the 1702 declared in split.tensors.count, for each quant.

The two candidate checkpoints are structurally IDENTICAL: same 276 expert tensors, same 96.92 %, same six other encodings at the same counts. Only the expert encoding differs.

Checkpoint	Expert encoding	bpw	Size	Was
UD-Q1_0	IQ1_XXXS, ggml 66	1.1875	370 GiB	in no upstream llama.cpp
UD-IQ1_S	IQ1_S, ggml 19	1.5625	473 GiB	upstream, absent here

Everything else (Q5_K, Q6_K, Q4_K, Q8_0, Q2_K, F32) was already served. An earlier draft recorded Q2_K as missing too; that came from a stale header comment listing six encodings, and reading the dispatch showed it has been served all along. Support status in this PR is read from the code.

The fork oracle, and what it is not

Type 66 is defined by no upstream llama.cpp: our pin 237ad9b96 ends at GGML_TYPE_Q1_0 = 41, and ggml-org master ad1de39e0 (2026-08-15) at GGML_TYPE_Q2_0 = 42. It exists only on unslothai/llama.cpp branch iq1-narrow.

This PR first refused that checkpoint on policy grounds, then admitted it on developer direction. The refusal rested on a policy premise rather than a technical one, and a recorded pin answers it. So unslothai/llama.cpp joins the oracle table as llama-cpp-unsloth, pinned at 36fe8e1cc, scoped to the sub-IQ1_S encodings alone. It never outranks vLLM or upstream llama.cpp and is a mirror source for one encoding family, not a design reference.

It is pinned gateable = no. AGENTS.md admits an oracle as gateable only once it demonstrably builds and runs the model, and neither has happened here: the port is grounded in the fork's SOURCE, read and cited, which is weaker than a running comparison. Running it needs the full 370 GiB checkpoint and, per Unsloth's own documentation, at least 450 GB of RAM. #933 owes that measurement, so the ungateable lane is visible debt.

The identification does not rest on trust: the bits per weight derived independently from GGUF offset deltas, before the fork was located, came out at 1.1875, matching the fork's own declaration exactly.

Review notes, for the parts worth arguing about

The grid cannot be validated by the obvious test. Both the vec_dot and the dequantizer read the same codebook, so a mistranscribed table moves both sides together and they still agree. Consistency is not correctness. Both tables are therefore extracted mechanically from their pinned blobs and sealed with a digest plus a lane census. That census earned its keep immediately: this branch first asserted the IQ1_S codebook was +/-1, and the table disproved it. It is TERNARY (6649 zeros, 4860 +1, 4875 -1), and five comments were wrong until the assertion failed. The seal binds harder for IQ1_XXXS, because a BRANCH can be rebased under its own name.

One bound was widened, and here is the reasoning rather than an assertion. The MatmulBTQuant NMSE case gained a per-case ceiling of 2e-3 for both new types. That case does not bound decode at all: it compares a Q8_K-quantized activation against an f32 reference, so it measures activation error, and Q8_K carries ONE scale per 256 elements. Uniformly random scale fields spread neighbouring 32-groups over a 15x range that no encoder emits, measured at 3.8e-4 against 3.8e-5 for a bounded spread. The synthetic scale was narrowed for the same reason the MXFP4 exponent already is, and the residual sits at 5.2e-4 on the thinnest shape, where m=4 and n=1 is four dot products. Mutation says where the empty band is: the weakest decode defect reads 2.7e-2, then 3.0e-1, 9.4e-1, 5.7, 5.7e+1. Decode stays bounded exactly by vec_dot versus an independent f64 dequantize-then-dot, and by MatmulBTQuant versus per-row vec_dot.

Evidence

• RED first, both encodings: the census test failed on ggml 19 and on ggml 66 alone, other five encodings passing, 10 of 12 assertions green each time.
• 19 mutations, all caught, each reported as applied/compiled/caught so a mutant that failed to build could not read as a pass. Tree restored byte-for-byte after each. The geometry and reader mutations are caught by the traits cross-check and keep-quant routing rather than by the numerics, which is what those cross-checks exist for.
• Gate: 485/485 ctest, 0 failed, re-run after merging origin/main.
• Coverage is not claimed from a grep of a passing log, which proves nothing because doctest prints captures only on failure: test_ops_quant_dot goes 184089 to 203611 assertions across 24 to 25 cases.

Owed, named rather than left to be discovered

The CUDA arm for both encodings (W10, W13). cuda_quant_dot.cu:1531 maps an unknown weight dtype to return false, which is a SILENT CPU fallback: a 2.4 T model would emit correct tokens at CPU speed and look healthy, and no token gate can see it. Listed in the spec.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

[localai-bot]

Merged on LOCAL evidence, not CI

CI is saturated, so the gate below was run locally on the exact merged tree (rebased onto c2019b0e3, linear, no merge commits). Recording what was and was not covered, so the substitution is auditable rather than implied.

Run here, green:

Check	Result
ctest full suite	485/485 passed, 0 failed (2 pre-existing skips)
agent-preflight.sh	75 gates ok
check-commit-trailers / check-commit-style	pass
check-oracle-pins / check-agent-record	pass
ASan + UBSan on both new decoders	ALL CLEAN
Mutation, IQ1_S	9/9 caught
Mutation, IQ1_XXXS	10/10 caught
Real-checkpoint decode vs independent transcription	1179648 weights bit-identical

The sanitizer run is a targeted substitute for CI's sanitize-cpu. The whole-project ASan build fails on a PRE-EXISTING GCC -Werror=array-bounds false positive at laguna_weights.cpp:156 (p.mlp_only_layers = {0}), unrelated to this change, so the two new decoders plus two already-shipped controls (IQ2_S, Q6_K) were compiled and run under -fsanitize=address,undefined directly, on EXACT-size heap buffers so an overrun is detectable rather than absorbed by slack.

That harness earned its controls immediately: the first run reported a Q6_K NaN, which was the harness writing the f16 delta at offset 0 for every type when q6_K trails it at 208. A control failing is how that got caught rather than being read as a Q6_K defect.

One test needed a note. test_cpu_x86_llamacpp_floor fails under box load and passes when idle (18.3 s, 10/10). It was control-proven on clean main at 5a0ffe9e3 under the same load, where it fails identically, so it is environmental and not this change.

Not run locally, and honestly outside what this box can do: windows-msvc-*, build-test-cpu-arm64, build-test-vulkan (llvmpipe only here), and cuda-fat-build (no nvcc on this host). This change is CPU-only decode plus records, so none of them exercise its new code paths, but that is a reason they are low risk, not a claim they ran.