feat(MODEL-MUSIC-MUSIC3): the queue its own comment named — a partial device arm, measured on Jetson Thor (#672)

[localai-bot]

FOLLOWING_AGENTS_PROTOCOL

src/vllm/model_executor/models/minimax_music3_speech.cpp:492 built its vt::Queue from a compile-time constant CPU device, under a comment that already named the seam:

CPU is what W2 ships and what every gate for this row has been taken on; a device arm is a queue, not a fork.

That was true, and it was unreachable — nothing could supply a different queue, so a 28.5 GB music model was host-only whatever hardware the box had.

It is now that queue.

What moves, and what does not

One of six stages moves. Saying so is the point: the alternative is a GPU claim that is really a CPU path with a flag set.

stage	--speech-device 1 runs it
8.6B Qwen3ForCausalLM — prefill, every decode step, its paged KV	device
guided logits, top-k draw, frame feedback	host (two 200 000-wide rows/step; not the cost)
0.646B RVQ depth decoder	host scalar loops — OWED
condition mix, 2.4B fp32 DiT, scheduler	host scalar loops — OWED
DAC Flow-VAE vocoder	host — BLOCKED, not merely unfinished

The language model reaches the device because it already rides the shared Qwen3DenseModel::ForwardEmbeds that five text registrations use; no model was forked. The vocoder is blocked on a missing op rather than on effort: vt has no ConvTranspose1d of any kind, and vt::Conv2d / vt::DepthwiseConv1d are registered CPU-only (src/vt/cpu/cpu_conv2d.cpp:111, src/vt/cpu/cpu_conv1d_depthwise.cpp:95), so the stage that is 88.5 % of the acoustic half has no CUDA kernel behind any op it could route through. Hand-rolling one outside the seam is what AGENTS.md forbids, so it is recorded as owed (spec §11.4) — with three consumers (vocoder1d is shared with MiniMax-H3 and the LTX-2 audio VAE), that op is its own row.

Three things this had to get right

1. The device selector is a MAPPING, not a cast, and it went on the SEAM. static_cast<vt::DeviceType>(device) is the defect minimax_h3_video.cpp:230-237 records. The tree already carried two copies of the correct three-question resolution (minimax_h3_video.cpp:255, ltx2_video.cpp:706); a third is where copies start to disagree, so this is multimodal::SpeechEngineDeviceType, keyed on the family string.
2. Zero is CPU, deliberately. Not vllm_model_params.device's 0=auto / 1=cpu / 2=cuda: auto would move every zero-filled caller — which is every caller written before this — onto an ungated path on an accelerator build. VideoModelParams::device already fixed this polarity for the sibling generative seam.
3. The paged KV had to move with the forward, and this is the piece that would have failed silently. dense_attn::KvSlice (dense_attn_block.h:233) labels the cache pointer with d.q.device, so 36 layers of host std::vector<uint16_t> on a CUDA queue is a host pointer wearing a device tensor's label. It is now allocated and zeroed on the queue's device.

Two things did not need doing, and both were checked rather than assumed: ResidentWeight stores its device copy on the OwnedTensor itself (w.d_dev), so weights upload once and are freed with the AR scope rather than cached against a reusable host address; and ForwardEmbeds already owns its own H2D/D2H.

Measured — Jetson Thor sm_110, in the container

Host kairos-4db2, aarch64, 14 cores, ~122 GB unified, driver 595.78, image vllmcpp-thor:cuda13.0.1, --runtime=nvidia -e NVIDIA_DISABLE_REQUIRE=1, -DVLLM_CPP_CUDA=ON -DVLLM_CPP_CUDA_ARCHITECTURES=110 -DVLLM_CPP_TRITON=OFF, no cutlass. Checkpoint mounted read-only from the NAS. Every run held flock $HOME/gpu.lock; one checkpoint resident at a time (this box reboots instead of OOM-killing); uptime recorded on both sides of each run.

Correctness — both arms, same box, same binary, SAME BOUNDS

tests/parity/test_minimax_music3_llm_real.cpp now takes VLLM_CPP_MUSIC3_DEVICE (default 0 = CPU, so an unset environment reproduces every number it ever printed), resolved through the same shared helper the engine calls so the gate cannot bind a different device than the engine. 25 teacher-forced steps vs frame_hiddens[:, :4096], 102 400 values:

arm	bit-identical	mean|d|	outside 2 bf16 ULP	golden-code mean rank	negative control
Thor CPU	9337 (9.118 %)	1.76348e-02	37 572 (36.69 %)	2.48	mean|d| 0.802531, 98.20 % outside
Thor CUDA	9324 (9.105 %)	1.71668e-02	36 509 (35.65 %)	2.44	mean|d| 0.802583, 98.18 % outside
control (torch sdpa_kernel(MATH), recorded)	12 036 (11.75 %)	1.475e-02	29 968 (29.27 %)	—	—

Both 4 cases / 220 assertions / 0 failed, no tolerance widened. The CUDA arm is marginally closer to the golden and both sit inside the measured torch-vs-torch control. The negative control fires identically on both, so the bound still discriminates rather than having gone slack. And the Thor CPU arm reproduces the recorded x86-64 numbers value for value, so the CPU path is unchanged across two architectures, not merely on the box that measured it.

Speed — both arms, same request

request	AR frames	delivered	--device 0	--device 1	ratio
--duration 0.1	2	0.070 s	835.1 s	846.6 s	1.014x — SLOWER
--duration 0.4	10	0.395 s	1512.1 s	1430.4 s	0.946x

Every other stage is the same host code on both arms and cancels in the difference:

D(2) = +11.5 s    D(10) = -81.7 s
slope = -11.65 s per AR frame    intercept = +34.8 s    break-even ~3 frames

Two points determine a line exactly, so that is an attribution with no residual and no error bar, stated as a fit rather than as a bound. The fixed cost is consistent with the one-time 17.2 GB ResidentWeight upload plus context creation — not measured separately, and offered as the plausible reading.

No parity claim. SGLang-Omni is still gateable = no; every reference axis in docs/BENCHMARKS.md stays PENDING.

Honest limits

• The two arms produce different songs, structurally. Spec §5 withdrew the token gate because the AR codes are a seeded torch.multinomial draw, so a different logit changes the drawn code and everything after it. No waveform comparison is offered; the LM parity gate above is the device arm's only numeric gate.
• One instrument defect, found inside this change. The gate's arm banner printed ran on '1' (VLLM_CPP_MUSIC3_DEVICE=1) on a CPU-only build with the variable unset — both fields collapsed inside doctest's MESSAGE chain. Had it not been read, the CPU arm's numbers would have been recorded as the device arm's. Rebuilt as one std::string.
• Owed and named, not discovered later (spec §11.4): the depth decoder and DiT through vt::MatmulBT with device-resident weights; vt::ConvTranspose1d with a CUDA provider for the vocoder; and the bit-identical row-wise parallelisation of LinearNoBias and vocoder1d::ConvTranspose1d, which needs its own measurement and is not in this change.

Gates

Local x86-64 CPU build, all green with non-zero assertion counts (assertions: 0 is a skip wearing a pass): test_speech_engine 11/38, test_capi 65/653, test_minimax_music3_speech 9/223, test_minimax_music3_loader 21/1393 (21/1413 with the checkpoint), test_minimax_music3_ar 25/338, test_minimax_music3_acoustic 27/265, test_minimax_music3_quant 29/125, test_minimax_music3_ar_real 4/894, test_minimax_music3_acoustic_real 6/76, test_minimax_music3_quant_real 6/319, test_minimax_music3_llm_real 4/220, test_minimax_h3 79/57395, test_indextts2_family 7/22, test_openai_api_server 62/733, test_speech_api 6/67.

Thor CUDA build: test_speech_engine 11/37 (one fewer BY DESIGN — the device-1 case takes its GRANTED branch there and its REFUSED branch on a CPU-only build, and prints which), test_minimax_music3_speech 9/223, test_minimax_music3_llm_real 4/220 on both arms.

Known-red, checked against a matched arm rather than assumed:

• test_capi on the Thor CUDA build. A control built in the same container from git archive origin/main c07526aa1 reproduces it exactly: the :487 SIGSEGV already recorded in docs/STATUS.md and in the Thor baseline sm_110 (Jetson Thor) ctest baseline at 2daa3287f: 15 red of 485, gated on (name, mode) pairs rather than a count #955, plus two structured_choice failures at :910/:932 that no baseline carried, because the crash aborts the run before them. Main 62/60 passed/587 assertions/2 failed; this tree 64/62/622/2 — the delta is exactly the two cases and 35 assertions this PR adds. Filed as test_capi: two structured_choice cases fail on a CUDA build, hidden behind the :487 SIGSEGV (#955 baseline understates test_capi) #994 rather than left behind a known-red name.
• check-env-doc REDs preflight on every branch on three VT_MOE_EXPERT_STREAM knobs that landed in 3005447f8; reproduced on pristine origin/main. Already filed as check-env-doc REDs preflight on every branch: three VT_MOE_EXPERT_STREAM knobs landed undocumented in 3005447f8 #995 / main is red on check-env-doc: three VT_MOE_EXPERT_STREAM vars are neither documented nor allowlisted #1000, not this row's.
• windows-msvc-* is windows-msvc-cpu/vulkan red on every PR based on c7cb59fbb: C4244 double->float in ltx2_video.cpp's StreamState<->Ltx2LatentState positions copy (#964) #968 (an LTX-2.5 C4244), untouched here.

scripts/agent-preflight.sh --staged --no-require-role exits 0 apart from the pre-existing check-env-doc red above.

Issue: #672
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]

[localai-bot]

CI, complete: 17 pass / 3 fail, and all three reds are attributed to a matched arm rather than to a label

job	result
build-test-cpu 30m53s, build-test-cpu-arm64 10m52s, build-test-vulkan 6m2s	pass
cuda-fat-build 1h36m24s	pass
sanitize-cpu (address,undefined) 44m0s, sanitize-cpu (thread) 51m41s	pass
verify (cpu) 9m56s, verify (vulkan) 9m12s	pass
commit-protocol-tag, documentation-checkpoint, pr-size, device-leakage, cuda-arch-features, vulkan-spirv-freshness, last-gated-commit, plan, build	pass
agent-record	fail — NOT this row
windows-msvc-cpu / windows-msvc-vulkan	fail — NOT this row

agent-record fails on one line and one only:

ERROR: production env var(s) read from src/+include/ are neither documented in
docs/ENVIRONMENT.md nor on scripts/env-doc-allowlist.txt:
  - VT_MOE_EXPERT_STREAM
  - VT_MOE_EXPERT_STREAM_SLOTS
  - VT_MOE_EXPERT_STREAM_SLOT_BYTES

Reproduced by running scripts/check-env-doc.py against a git archive of pristine origin/main: identical three variables. They landed in 3005447f8 and are already filed twice, as #995 and #1000. This row reads exactly one new environment variable, VLLM_CPP_MUSIC3_DEVICE, and it is read from tests/ — outside the src/+include/ scope this checker covers, and consistent with its sibling VLLM_CPP_MUSIC3_CHECKPOINT.

windows-msvc-* is #968 (the LTX-2.5 C4244). Matched arm: the same two jobs are FAILURE on #996 and #992, neither of which touches the speech surface. Nothing here touches LTX2.

Both were checked against a matched arm rather than read off a known-red list, because a known-red list tells you a job is often red, never that today's red is the same one — which is how the two structured_choice failures in #994 stayed invisible behind the test_capi SIGSEGV for as long as they did.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]