mudler · bakon11 · Aug 13, 2026 · Aug 13, 2026 · Aug 17, 2026
diff --git a/.agents/specs/gemma4-rocm-fp8-moe.md b/.agents/specs/gemma4-rocm-fp8-moe.md
@@ -58,3 +58,85 @@ and env-gated MoE paths do not. Revisit as a **separate** PR with CUDA golden.
 
 `VT_GEMMA4_MLP_MOE_PARALLEL` is documented but **not** wired in this tip (was only
 in the deferred layer-loop path).
+
+## Measured KEEP (2026-08-13, contributor lab)
+
+Hardware: 2x R9700 gfx1201, ROCm 7.2.4, kernel 7.0.0-29-generic.
+Model: Gemma-4-26B-A4B-it FP8. Fair protocol: `PREFIX_CACHE=0` + unique pads.
+
+Recipe as run: SharedK-WMMA on, FLASH/FMHA/scoreless off,
+`VT_GEMMA4_PREFILL_GEMM_M=2048`, `VT_GEMMA4_PREFILL_PEER_ACT=1`, batch MoE
+`T>=64`, decode `KV_SPLITS=16` / `SLIDE_SPLITS=8` / `SLIDE_WARPS=16` /
+`SPLIT_WARPS=12`. Speculative, ngram, layer-split: **off**.
+
+**Four of those names are not read by any product code in this tree, so this
+recipe is not reproducible as written.** `VT_ATTN_DECODE_KV_SPLITS`,
+`VT_ATTN_DECODE_SLIDE_SPLITS` and `VT_ATTN_DECODE_SPLIT_WARPS` appear only in
+`tests/vt/test_gemma4_rocm_fp8_seams.cpp`, and `VT_ATTN_DECODE_SLIDE_WARPS`
+appears nowhere at all; `git grep` over `src/` and `include/` returns zero hits
+for all four. That is [#845](https://github.com/mudler/vllm.cpp/issues/845): the
+seam test asserts `EnvInt(name, 16) == 16` with the variable unset, a tautology
+that passes whether or not the knob exists, which is how three names that
+nothing reads came to look real. The first two knob names above were also
+abbreviated and are spelled here as the product spells them
+(`PEER_ACT` is `VT_GEMMA4_PREFILL_PEER_ACT`, `gemma4_moe.cpp:1026`;
+`PREFILL_GEMM_M` is `VT_GEMMA4_PREFILL_GEMM_M`, `:1016`, and `2048` is its
+default, not an override).
+
+This file's `**Status:**` line records the run as `PR tip
+feat/gemma4-rocm-fp8-split` with **no commit SHA**, so which tree produced these numbers cannot be
+established from this repository. Either the four decode splits were live on that
+tree and it is not this one, or they were inert and the numbers were produced by
+the shipped defaults. Owed to the contributor: the tree SHA, and which of the two
+it was. Until then the decode figure in particular has no recipe behind it, and
+`docs/USAGE.md` publishes only the knobs a reader can actually set.
+
+| Depth | median prefill t/s | notes |
+|------:|-------------------:|-------|
+| ~3k | 2112 | 3 reps, Paris |
+| ~11k | 2014 | first rep 1170 outlier dropped from claim; median 2014 |
+| ~18k | 1705 | 3 reps |
+| ~42k | 1099 | 2 reps |
+
+Decode stream 64 tok: **55.5 t/s** temp=0, **49.1 t/s** temp=0.7.
+Quality: Paris, arith `63`, `gemma4` parser `tool_calls`, `/health` `/metrics`.
+No engine-fatal / hipError in the closeout log.
+
+Same-box Vulkan Q8 unique-pad bar: **3503 @11k / 2714 @42k**. KEEP gap ~1.74x / 2.47x.
+This row ships the reliable ROCm plateau. It does **not** close that bar.
+
+### Rejected (do not default-on)
+
+FMHA_WMMA2 (quality fail + 0.67x @11k). Isolated P1 cm1 wg256 (~1.13x KEEP; need ~3.35x isolated). Layer-split FIFO (~0.60x vs the Q8 bar). Head-TP peer-read (peer BW). hipBLASLt dual-GPU (fatal or slower).
+
+### Residual compiler / ACO
+
+Faithful HIP cm1 hsaco on gfx1201: 192 VGPR, 339 spills, 940 B private, vs KEEP SharedK 35 spills / 120 B. Output-slicing bottoms out at 235 spills / 736 B (D_PT=1). Mechanism of the isolated 1.13x, not by itself a named LLVM defect.
+
+Matched RADV/ACO (Mesa 26.0.3, discrete R9700, llama.cpp coopmat1 spec Br=16 Bc=64 wg=128 sg=32 row_split=4): official pipeline stats are **0 spilled VGPR, 0 scratch** at VGPR=256 (f16 and q8_0 d=512). That is an (a)-leaning HIP-LLVM vs ACO disparity on equivalent ownership. Smallest LLVM/rocWMMA component is not named. Phase-2 rebuild stays blocked until that component and a measurable prediction exist. A general codegen fix, if one exists, is a separate claim from the 3.35x bar.
+
+## Owed
+
+- **A vLLM-ROCm denominator for every number in "Measured KEEP".** They are
+  engine-side figures with nothing on the other side, so under AGENTS.md "Gates"
+  they are not a throughput result at all, and `docs/BENCHMARKS.md` keeps this
+  backend at `PENDING: no binding throughput number`. The oracle is not
+  hypothetical: `docs/ROCM.md` §5 documents two working Docker vLLM-ROCm recipes
+  on this hardware family, tier 2 building this project's pinned commit
+  `555967922` inside `rocm/vllm-dev:base` in about 6.5 minutes. Owed: the same
+  model, quantization, prompt set, request shape, concurrency and cache policy on
+  both sides of one idle 2x R9700 box, and the ratio per depth. Only the
+  contributor has that hardware.
+- **The tree SHA behind the 2026-08-13 run**, and whether the four decode splits
+  named in the recipe were live on it. See the note under "Measured KEEP" and
+  [#845](https://github.com/mudler/vllm.cpp/issues/845).
+- **`.agents/benchmark-record.md` is deliberately not appended here.** It is the
+  append-only measurement log, and a number whose tree is unidentified and whose
+  denominator is missing must not enter it: once logged, a figure gets quoted as
+  measured. It goes in when the two items above are discharged.
+- **The Vulkan Q8 comparison stays open, not closed.** Same box, ~1.74x at 11k
+  and ~2.47x at 42k in Vulkan's favour. AGENTS.md forbids reading that as a
+  ceiling. The next traceable hypothesis is the one this file already names: the
+  HIP-LLVM versus ACO register-allocation disparity, 339 spills against ACO's 0,
+  and it needs the smallest LLVM/rocWMMA component named before a phase-2
+  rebuild.
diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md
@@ -229,7 +229,7 @@ on CUDA/CPU builds beyond the documented behavior.
 | `VT_GEMMA4_PREFILL_BATCH_MOE` | auto / `1` in lab recipe | `=1` group-by-expert prefill GEMM for `T>=64`; `=0` serial M=1 (slow). Unset = auto |
 | `VT_GEMMA4_MLP_MOE_PARALLEL` | off | `=1` run Gemma4 MLP and MoE on two HIP streams (lab; wall ~flat on R9700). Not wired in this PR tip (decode-graph-free split) |
 | `VT_ATTN_PREFILL_FLASH` | off | `=1` SGLang-style BM×BN GQA flash prefill (lab A/B) |
-| `VT_GEMMA4_PREFILL_GEMM_M` | `256` | Tokens per expert in prefill-batch GEMM chunks (`16..2048`). Larger M → fewer launches; lab `512` ~+37% prefill vs `64` |
+| `VT_GEMMA4_PREFILL_GEMM_M` | `2048` | Tokens per expert in prefill-batch GEMM chunks (`16..8192`; out-of-range values are ignored and the default is used). Larger M → fewer launches; lab `512` ~+37% prefill vs `64`, and `512`→`2048` ~+80 eng @11k vs the WMMA baseline (2026-08-10), which is why the default is `2048`. Lab KEEP on dual R9700 uses the default |
 | `VT_GEMMA4_HOST_EXPERT_MB` | `512` | Host-side expert staging budget (MiB) for non-resident paths |
 | `VT_GEMMA4_LAYER_TRACE` | off | `=1` layer GPU-synced phase timers; `=2` per-layer heartbeats |
 | `VLLM_CPP_HTTP_FIXED_POOL` | `1` (fixed) | `=0` reverts the HTTP worker pool to the legacy dynamic mode. Production uses the capacity-derived fixed pool; the opt-out exists for same-binary A/B attribution |

diff --git a/docs/ROCM.md b/docs/ROCM.md
@@ -9,8 +9,11 @@ Gemma-3-1B-it is 48/48 token-identical to two independent vLLM-ROCm oracles;
 Qwen3-0.6B exposed a deterministic cross-version near-tie. Qwen3.5-0.8B also
 runs all-native through the GDN stack, but its CPU/ROCm divergence is still an
 open correctness gap. The Gemma-4 FP8 MoE and SharedK-WMMA path has contributor
-runtime evidence on 2x R9700; this repository has only CPU link coverage for
-that path, and no matched oracle performance result.
+runtime evidence on 2x R9700 (KEEP fair ~2014 t/s @~11k / ~1099 @~42k,
+`PREFIX_CACHE=0`; decode ~55 t/s). This repository has only CPU link coverage
+for that path, and no matched vLLM-ROCm oracle performance result. Vulkan Q8
+prefill on the same box is faster (~3503 / ~2714); the ROCm path is the
+reliable recipe, not that bar.
 
 This page exists because several people offered hardware in
 [issue #41](https://github.com/mudler/vllm.cpp/issues/41), and it answers the

diff --git a/docs/USAGE.md b/docs/USAGE.md
@@ -3899,9 +3899,27 @@ ENVIRONMENT.md (`VT_GEMMA4_RESIDENT_*`, `VT_ATTN_*`). Defaults stay safe off RDN
 GetBlas keeps two per-thread hipBLAS handles (`tls_slots[2]`, device 1 → slot 1)
 so a 0→1 hop does not destroy GPU0's handle. `ProductGetBlasHandle` is the
 test accessor for that file-local `GetBlas`. HIP live probe is a separate CTest
-target (exit 77 if `HIP_VISIBLE_DEVICES` empty); it enters capture so production `StreamIsCapturing` is load-bearing. No new env. This PR does **not**
-restructure the Gemma-4 layer loop or enable decode hipGraph (those stay lab-only
-until a CUDA token-exact gate can land them).
+target (exit 77 if `HIP_VISIBLE_DEVICES` empty); it enters capture so production `StreamIsCapturing` is load-bearing. No new env.
+This path does **not** restructure the Gemma-4 layer loop or enable decode hipGraph
+(those stay lab-only until a CUDA token-exact gate can land them).
+
+Contributor KEEP recipe (2x R9700 gfx1201, ROCm 7.2.4, `PREFIX_CACHE=0`, unique
+pads, 2026-08-13): SharedK-WMMA on, FLASH/FMHA off, `VT_GEMMA4_PREFILL_GEMM_M=2048`
+(the default), `VT_GEMMA4_PREFILL_PEER_ACT=1` (the default), batch MoE `T>=64`.
+Fair median prefill **2014 t/s @~11k** and **1099 t/s @~42k**; stream decode
+**55 t/s** temp=0. Paris / arith `63` / `gemma4` tool_calls held. Speculative,
+ngram, FMHA, and layer-split are **out of this recipe**. Details:
+[spec](../.agents/specs/gemma4-rocm-fp8-moe.md).
+
+These are contributor-lab numbers against **no denominator**: no pinned vLLM-ROCm
+run on the same box, same model, same quantization and same request shape exists
+for them, so `docs/BENCHMARKS.md` still records this backend as `PENDING: no
+binding throughput number` and this recipe does not change that. The decode
+figure is also not reproducible from the knobs above: the as-run recipe set four
+further decode splits that no product code in this tree reads
+([#845](https://github.com/mudler/vllm.cpp/issues/845)), and they are recorded in
+the spec rather than here, because a recipe on this page has to be one a reader
+can follow.
 
 ## LTX-2.5 text conditioning