Skip to content

fix(MODEL-MUSIC-MUSIC3): the e2e gate asked for 60 s of music, not 0.1 s — one JSON key, and a weight-load stall that was never real (#852, #925) - #942

Merged
localai-bot merged 3 commits into
mainfrom
row/MUSIC3-E2E-LOAD
Aug 15, 2026
Merged

Conversation

@localai-bot

Copy link
Copy Markdown
Collaborator

MiniMax-Music3 generates a song end to end, observed

POST /v1/audio/speech -> 200 audio/wav, 12332 bytes
request shape: 2 AR frames -> 6 latent frames -> 3072 samples/channel (0.069660 s)
waveform:      6144 int16 samples, 6144 non-zero, 0 clipped, range [42, 156],
               RMS 0.00267716, 2818 of 3072 positions differ between L and R

Re-read independently of the gate at build/music3/minimax_music3_e2e_http.wav:
44100 Hz, 2 channels, 16-bit, 3072 frames, 0.069660 s, peak 156, 67 distinct
left-channel values.

The defect is one JSON key

The request body posted "audio_duration_s": 0.1. That is the name of the
field (speech_api.h:55), not the wire key. ParseSpeechRequest reads
audio_duration (or duration) — what docs/USAGE.md:1235 documents and what
every other test in the tree uses. The key was dropped silently,
audio_duration_s stayed at its 0.0 sentinel, and Music3ResolveRequest
substituted kMusic3DefaultDurationSeconds = 60.0.

So the gate asked for a sixty-second song:

asked for actually ran
duration 0.1 s 60 s
AR frames 2 1500
denoise windows 1 8
vocoder latents 6 5167
samples/channel 3072 2645504

A ~750x job — and one that could never have passed anyway, because the case
asserts the payload length 0.1 s implies.

The recorded diagnosis was wrong, and how it was wrong generalizes

Four runs (85 min, 34 min, 1 h 49 m, and one of mine at 15 h 19 m) were
placed "inside LoadQwen3ForCausalLMWeights" on the evidence that the four
language_model/*.safetensors fds were open and the depth decoder's was not.

Neither fact says that. LoadBf16Direct borrows the mapping when it can
(BorrowStTensorBytes), and a borrowed OwnedTensor holds its own
shared_ptr to it — so those fds stay open for the whole request, long after the
loader returned. The depth decoder's tensors are copied, so its
SafetensorsFile dies and its fd closes on return. The fd pattern is a
copy-versus-borrow artifact, not a program counter.

What resolves it is a symbol-resolved profile — perf record -g on the live
process, 173K samples, symbols intact because the binary was not rebuilt under
it:

phase span signature
LM weight load 0 → 180 s 1 thread, state D, ~92 MB/s off the NAS, 3.7 s user CPU
AR generate 180 → 19154 s 20 threads, LinearNoBias 42-57%, Threadpool::Barrier 25%, Bt16Avx512 12-14%
acoustic 19154 s → killed at 15 h 19 m vocoder1d::ConvTranspose1d 88.5%, Conv1d 7.7%

The load is 3 minutes, in both binaries — the same ~3 minutes
test_minimax_music3_llm_real takes — and it is I/O bound, not spinning. The
__sched_yield frames in the earlier profile are Threadpool::PollForWork's
bounded hybrid poll around a genuinely compute-bound run.

Two supporting measurements, taken rather than argued: the 200000 x 4096
lm_head scalar transpose that was suspected costs 1.388 s, and the vocoder
at this request's real shape costs 5.4 s (against 84 s at the capture's
L=86).

The guard

A silently dropped knob is what made this invisible for four runs, so
audio_duration_s is now refused and named — like every other unsupported
field in that function, whose own comment already said ignoring one "returns a
200 carrying audio the caller did not ask for".

RED first: the two CHECK_THROWS in test_speech_api.cpp did not throw.

test_speech_api.cpp:144: ERROR: CHECK_THROWS( ... "audio_duration_s":0.1 ... ) did NOT throw at all!
test_speech_api.cpp:145: ERROR: CHECK_THROWS( ... extra_params ... ) did NOT throw at all!
[doctest] test cases: 1 | 0 passed | 1 failed | assertions: 21 | 19 passed | 2 failed

The guard is narrow on purpose: refusing every unknown key would break the
extra_params bag's forward compatibility, while this one near-miss is the
spelling the struct itself invites (#925).

Not changed

LinearNoBias keeps its reduction order and its -ffp-contract=off scalar
triple loop — W2/W3 gate it against torch. It is 42-57% of the AR profile and
parallelising it over output rows would not move a gated number, but that is a
separate, measured change with its own evidence. No speed number is claimed:
CPU only, dgx.casa down.

Gates

All on this box (20-core x86_64, CPU only), CASE and assertion counts:

gate cases assertions
test_minimax_music3_e2e_real (both env vars) 5 535
test_speech_api 5 56
test_minimax_music3_loader 21 1413
test_minimax_music3_ar 25 338
test_minimax_music3_ar_real 4 894
test_minimax_music3_acoustic 27 265
test_minimax_music3_acoustic_real 6 985
test_minimax_music3_speech 9 223
test_minimax_music3_llm_real 4 220
test_minimax_music3_quant 29 125
test_minimax_music3_quant_real 6 319
test_minimax_h3 79 57395
test_indextts2_family 7 22
test_openai_api_server 61 709
test_capi 63 617
test_qwen3_paged_engine 2 0 — SKIP, Qwen3-0.6B/4B snapshots are dgx-only

scripts/agent-preflight.sh --no-require-role is green except
test_cpu_x86_llamacpp_floor, the known load-dependent #618 (4 != 2,
NO_QUIET_WINDOW instead of GIVING_UP), which the same tree passed 25
minutes earlier at lower load.

Records

.agents/model-matrix.md, docs/FEATURES.md and docs/STATUS.md conflicted on
merge and were rebuilt from main's version with only my rows reapplied; every
unrelated key was diffed against origin/main and is byte-identical (2 rows, 2
rows and 1 row changed respectively). The tree was grepped explicitly for
conflict markers.

Issue: #852, #925, #672

🤖 Generated with Claude Code

mudler added 2 commits August 15, 2026 17:07
…1 s -- one JSON key, and a weight-load stall that was never real (#852, #925)

MiniMax-Music3 now produces a song end to end, over HTTP, observed:

  POST /v1/audio/speech -> 200 audio/wav, 12332 bytes
  2 AR frames -> 6 latent frames -> 3072 samples/channel (0.069660 s)
  6144 int16 samples, 6144 non-zero, 0 clipped, range [42, 156],
  RMS 0.00267716, 2818 of 3072 positions differ between L and R

  HTTP case alone   1 case  |  21 assertions | 0 failed |  7:54 wall
  whole file        5 cases | 535 assertions | 0 failed | 31:14 wall

The WAV was re-read independently of the gate: 44100 Hz, 2 channels, 16-bit,
3072 frames, 67 distinct left-channel values.

THE DEFECT IS ONE KEY. The request body posted `"audio_duration_s": 0.1`. That
is the name of the FIELD (`speech_api.h:55`), not the wire key.
`ParseSpeechRequest` reads `audio_duration`, which is what `docs/USAGE.md:1235`
documents and what every other test uses. The key was dropped silently,
`audio_duration_s` stayed at its `0.0` sentinel, and `Music3ResolveRequest`
substituted `kMusic3DefaultDurationSeconds = 60.0`.

So the gate asked for a sixty-second song:

                    asked for      actually ran
  duration          0.1 s          60 s
  AR frames         2              1500
  denoise windows   1              8
  vocoder latents   6              5167
  samples/channel   3072           2645504

A ~750x job, and one that could never have passed anyway, because the case
asserts the payload length 0.1 s implies.

THE RECORDED DIAGNOSIS WAS WRONG, AND HOW IT WAS WRONG GENERALIZES. Four runs
were placed "inside `LoadQwen3ForCausalLMWeights`" on the evidence that the four
`language_model/*.safetensors` fds were open and the depth decoder's was not.
Neither fact says that. `LoadBf16Direct` BORROWS the mapping when it can, and a
borrowed `OwnedTensor` holds its own `shared_ptr` to it, so those fds stay open
for the whole request long after the loader returned; the depth decoder's
tensors are COPIED, so its `SafetensorsFile` dies and its fd closes on return.
The fd pattern is a copy-versus-borrow artifact, not a program counter.

What resolves it is a symbol-resolved profile. `perf record -g` on the live
process, 173K samples, symbols intact because the binary was not rebuilt under
it:

  LM weight load   0 -> 180 s        1 thread, state D, ~92 MB/s off the NAS,
                                     3.7 s of user CPU in the whole phase
  AR generate      180 -> 19154 s    20 threads, LinearNoBias 42-57%,
                                     Threadpool::Barrier 25%, Bt16Avx512 12-14%
  acoustic         19154 s -> killed vocoder1d::ConvTranspose1d 88.5%,
                   at 15 h 19 m      Conv1d 7.7%

The load is THREE MINUTES, in both binaries -- the same ~3 minutes
`test_minimax_music3_llm_real` takes -- and it is I/O bound, not spinning. The
`__sched_yield` frames in the earlier profile are `Threadpool::PollForWork`'s
bounded hybrid poll around a genuinely compute-bound run. Two supporting
measurements, taken rather than argued: the 200000 x 4096 `lm_head` scalar
transpose that was suspected costs 1.388 s, and the vocoder at this request's
real shape costs 5.4 s against 84 s at the capture's L=86.

THE GUARD, because a silently dropped knob is what made this invisible.
`audio_duration_s` is now REFUSED and named, like every other unsupported field
in that function -- whose own comment already said ignoring one "returns a 200
carrying audio the caller did not ask for". Red first: the two `CHECK_THROWS` in
`test_speech_api.cpp` did not throw. The guard is narrow on purpose; refusing
every unknown key would break the `extra_params` bag's forward compatibility,
while this one near-miss is the spelling the struct itself invites (#925).

NOT CHANGED. `LinearNoBias` keeps its reduction order and its
`-ffp-contract=off` scalar triple loop; W2/W3 gate it against torch. It is
42-57% of the AR profile and parallelising it over OUTPUT ROWS would not move a
gated number, but that is a separate, measured change with its own evidence. No
speed number is claimed: CPU only, `dgx.casa` down.

GATES, all on this box, CASE and assertion counts:

  test_minimax_music3_e2e_real       5 |   535   (with both env vars)
  test_speech_api                    5 |    56
  test_minimax_music3_loader        21 |  1413
  test_minimax_music3_ar            25 |   338
  test_minimax_music3_ar_real        4 |   894
  test_minimax_music3_acoustic      27 |   265
  test_minimax_music3_speech         9 |   223
  test_minimax_music3_quant         29 |   125
  test_minimax_music3_quant_real     6 |   319
  test_minimax_h3                   79 | 57395
  test_indextts2_family              7 |    22
  test_openai_api_server            61 |   709
  test_capi                         63 |   617
  test_qwen3_paged_engine            2 |     0   SKIP: Qwen3-0.6B/4B snapshots
                                                 are dgx-only, absent here

`scripts/agent-preflight.sh --no-require-role` is green except
`test_cpu_x86_llamacpp_floor`, which is the known load-dependent #618 (`4 != 2`,
`NO_QUIET_WINDOW` instead of `GIVING_UP`) and which the SAME tree passed 25
minutes earlier at lower load.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]
Issue: #852, #925, #672
# Conflicts:
#	.agents/model-matrix.md
#	docs/FEATURES.md
#	docs/STATUS.md
@localai-bot
localai-bot merged commit 2688e65 into main Aug 15, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants