MiniMax-Music3 e2e: the LM weight load spins for hours in the SpeechEngine path but takes ~3 min in the LLM gate (#672)

[localai-bot]

The defect

test_minimax_music3_e2e_real's full-tail and HTTP cases never complete. Both
sit inside LoadQwen3ForCausalLMWeights effectively forever, while the same
weights load in ~3 minutes in test_minimax_music3_llm_real.

This is a performance defect, not a correctness one — every stage is gated
and green (W1-W7). It is the single thing standing between MiniMax-Music3 and an
observed end-to-end song.

Evidence, measured on two independent runs

The implementer stopped runs at 85 min and 34 min. I then ran it myself
and stopped at 1 h 49 m wall / 2 h 41 m CPU. All three runs show the same
signature:

signal	value
open fds	the four language_model/model-0000{1..4}-of-00004.safetensors, and nothing else
depth-decoder file	never opened
RSS	flat at 18.2 GiB (17.38 GiB on the implementer's run)
CPU	147 % across 20 threads — i.e. ~1.5 cores busy, 18-19 threads idle-spinning
state	R (running) — burning CPU, not blocked on I/O

perf record on the live process (8905 samples) shows the spin directly:

--3.55%--__sched_yield
    --2.83%--entry_SYSCALL_64_after_hwframe
        --2.80%--do_syscall_64  →  __x64_sys_sched_yield  →  do_sched_yield  →  schedule

with two dominant unresolved user-space hotspots (~44 % and ~29 %). Symbols did
not resolve because the binary had been rebuilt under the running process
((deleted)), so the exact frames are still owed.

It is not the mount: a sequential cat of the same 17.2 GB completes in
20.6 s.

Why this looks like a threadpool barrier

One thread does work while the rest spin on sched_yield. That is consistent
with the related measurement recorded in b5a5f3b18: the vt threadpool is
idle for the majority of a request's wall clock, because the RVQ depth
decoder and the DiT do not go through vt at all, and LinearNoBias is a scalar
triple loop under -ffp-contract=off (deliberately, so W2/W3 could gate its
reduction order against torch).

The diagnostic lever

The two paths differ and both load the same shards:

• test_minimax_music3_llm_real — ~3 min
• the SpeechEngine / e2e path — hours

Whatever differs between them is the bug. A plausible first hypothesis is a
dtype conversion (e.g. a scalar bf16→f32 widening of ~8.6 B parameters) taken on
one path and not the other, but that is a hypothesis and not a finding.

Done when

test_minimax_music3_e2e_real with VLLM_CPP_MUSIC3_CHECKPOINT and
VLLM_CPP_MUSIC3_DIT=1 completes and is observed to pass, producing a
44100 Hz stereo WAV from an HTTP request — with the load time recorded.

Note for whoever picks this up

That test silently skips without both env vars: it reports
5 cases passed | 0 assertions with neither set. A gate that cannot say how many
things it examined has not reported.

[localai-bot]

Correction to the evidence above — the fd signal was mine and it was wrong

The issue body treats "only the four language_model shard fds are open, the
depth file is never opened" as evidence that the process is inside
LoadQwen3ForCausalLMWeights. That inference does not hold: the loader mmaps
the shards, and a borrowed mmap keeps its fd open long after the load returns. So
the fd set is an artifact of mapping lifetime, not a position in the call stack.

Retracting it explicitly, because it points a diagnosis at the wrong function and
someone would have spent hours inside a loader that had already finished.

What still stands, unchanged:

• three runs never completed (85 min, 34 min, and 1 h 49 m wall / 2 h 41 m CPU)
• RSS flat at ~18.2 GiB, state R (running) — burning CPU, not blocked on I/O
• 147 % CPU across 20 threads — ~1.5 cores busy while the rest spin
• perf shows __sched_yield → do_sched_yield → schedule — a spin-wait —
  with two dominant unresolved user-space frames (~44 %, ~29 %)
• cat of the same 17.2 GB completes in 20.6 s, so the mount is not the cause
• the same weights load in ~3 min in test_minimax_music3_llm_real

What is now open again: where the process actually is. The two-path
comparison is still the right experiment, but it must be driven by a symbol-resolved
profile, not by fd inspection. The earlier profile resolved nothing because the
binary was rebuilt under the running process ((deleted)).

A better host is available

192.168.68.23 (Thor, aarch64) — I previously wrote it off as unusable, and that
was also wrong: I had only checked / and non-sudo nvidia-smi.

disk	363 GB free on /usr/local (not /, which is a 4.4 GB loop)
checkpoint	NAS mounted at /usr/local/nas_share/checkpoints/minimax-music3
memory	122 GB
cores	14
load	~3, versus ~111 on the shared x86 box
GPU	works via sudo -n nvidia-smi
toolchain	g++ 13.3.0, git; cmake 4.4.2 + ninja installed at /usr/local/tools/py/bin (needs PYTHONPATH=/usr/local/tools/py)
no CUDA toolkit	no nvcc anywhere, so a CUDA build is not available there yet

The contention difference matters on its own: one earlier attempt was invalidated
by a full ctest -j6 running against the same 18.5 GB working set.

[localai-bot]

Root-caused and fixed: PR #942. It was never a weight-load stall.

The gate now passes: POST /v1/audio/speech -> 200 audio/wav, 12332 bytes, a
44100 Hz 16-bit stereo WAV of 3072 samples/channel. 5 cases | 535 assertions |
0 failed for the whole file with both env vars set.

The defect is one JSON key. The body posted "audio_duration_s": 0.1 — the
name of the FIELD (speech_api.h:55), not the wire key. ParseSpeechRequest
reads audio_duration, which is what docs/USAGE.md:1235 documents and what
every other test uses. It was dropped silently, so Music3ResolveRequest
substituted kMusic3DefaultDurationSeconds = 60.0 and the gate asked for a
sixty-second song: 1500 AR frames instead of 2, 8 denoise windows instead of
1, 5167 vocoder latents instead of 6. A ~750x job, which would then have failed
its own payload assertion anyway. The near-miss key is now REFUSED (#925).

On the fd evidence — you retracted it, and here is what replaces it.
LoadBf16Direct borrows the mapping when it can, and a borrowed OwnedTensor
holds its own shared_ptr to it, so the LM shard fds stay open for the whole
request; the depth decoder's tensors are copied, so its fd closes on return.
The pattern is copy-versus-borrow, not a program counter.

The symbol-resolved profile (perf record -g, 173K samples, binary not
rebuilt under the process):

phase	span	signature
LM weight load	0 -> 180 s	1 thread, state D, ~92 MB/s off the NAS, 3.7 s user CPU
AR generate	180 -> 19154 s	20 threads, LinearNoBias 42-57%, Threadpool::Barrier 25%, Bt16Avx512 12-14%
acoustic	19154 s -> killed at 15 h 19 m	vocoder1d::ConvTranspose1d 88.5%, Conv1d 7.7%

Your two unresolved frames map to LinearNoBias (~44%) and
Threadpool::Barrier (~29%). The __sched_yield is
Threadpool::PollForWork's bounded hybrid poll around a genuinely
compute-bound run, not a stall.

The load is 3 minutes in BOTH binaries — I watched RSS climb 0 -> 17.37 GiB
in 180 s at 92 MB/s in state D, then the process flip to 20 threads and start
burning CPU. test_minimax_music3_llm_real is 8:34 wall total for load plus 25
teacher-forced steps. Two more measurements that close the old hypotheses: the
200000 x 4096 lm_head scalar transpose costs 1.388 s, and the vocoder at
the request's real shape costs 5.4 s.

So the honest summary is that nothing was ever hung. Every run was killed part
way through a job 750x larger than intended.

[localai-bot]

Resolved — and the title of this issue is wrong. There was never a load stall.

The cause is one JSON key. The e2e test posted "audio_duration_s" — the name
of the field (speech_api.h:55) — but ParseSpeechRequest reads
audio_duration, which docs/USAGE.md:1235 documents and every other test uses.
The key was silently dropped and Music3ResolveRequest substituted
kMusic3DefaultDurationSeconds = 60.0:

	asked for	actually ran
duration	0.1 s	60 s
AR frames	2	1500
denoise windows	1	8
vocoder latents	6	5167
samples/channel	3072	2 645 504

A ~750× job, which would have failed its own payload assertion even if it had
finished. Every run — the 85 min, the 34 min, and my 1 h 49 m — was killed
part-way through a job three orders of magnitude larger than intended.

The resolved profile (173 K samples, binary not rebuilt under the process)

phase	span	signature
LM weight load	0 → 180 s	1 thread, state D, ~92 MB/s off the NAS
AR generate	180 s → 19 154 s	LinearNoBias 42-57 %, Threadpool::Barrier 25 %
acoustic	19 154 s → killed	ConvTranspose1d 88.5 %, Conv1d 7.7 %

So the LM load takes 3 minutes here too — identical to
test_minimax_music3_llm_real. The "two paths differ" premise this issue was
built on was an artifact of the 750× job, not a real divergence.

My unresolved ~44 % and ~29 % frames were LinearNoBias and
Threadpool::Barrier. The __sched_yield I read as a pathological spin is
PollForWork's bounded hybrid poll around compute-bound work — normal, not a
defect. Two hypotheses closed with numbers: the suspected 200000×4096 lm_head
transpose is 1.388 s, and the vocoder at the real shape is 5.4 s.

The fd inference, a second time

Beyond the mmap-lifetime point already retracted above, there is a sharper
reason: LoadBf16Direct borrows the mapping and a borrowed OwnedTensor
holds its own shared_ptr, so the LM fds stay open for the whole request; the
depth decoder's tensors are copied, so its fd closes on return. The fd set
distinguishes copy-from-borrow, not a program counter.

Observed

POST /v1/audio/speech -> 200 audio/wav, 12332 bytes
2 AR frames -> 6 latent frames -> 3072 samples/channel (0.069660 s)
6144 int16, 6144 non-zero, 0 clipped, RMS 0.00267716,
2818 of 3072 positions differ between L and R

44100 Hz, 2 ch, 16-bit, 3072 frames. test_minimax_music3_e2e_real: 5 cases /
535 assertions / 0 failed.

Two honest limits, also written into the spec: this is 0.07 s of audio, not a
song, and its samples are compared to nothing — §5 withdrew the token gate
because the AR codes are a seeded multinomial draw and the initial latents a
seeded randn_tensor. It proves the composition runs end to end and emits
well-formed stereo PCM of the implied length; the numeric parity lives in the
other four cases, which pass in the same run.

The silently-ignored request key is filed separately as #925.