fix(MODEL-MUSIC-MUSIC3): the e2e gate asked for 60 s of music, not 0.1 s — one JSON key, and a weight-load stall that was never real (#852, #925) - #942
Merged
Conversation
…1 s -- one JSON key, and a weight-load stall that was never real (#852, #925) MiniMax-Music3 now produces a song end to end, over HTTP, observed: POST /v1/audio/speech -> 200 audio/wav, 12332 bytes 2 AR frames -> 6 latent frames -> 3072 samples/channel (0.069660 s) 6144 int16 samples, 6144 non-zero, 0 clipped, range [42, 156], RMS 0.00267716, 2818 of 3072 positions differ between L and R HTTP case alone 1 case | 21 assertions | 0 failed | 7:54 wall whole file 5 cases | 535 assertions | 0 failed | 31:14 wall The WAV was re-read independently of the gate: 44100 Hz, 2 channels, 16-bit, 3072 frames, 67 distinct left-channel values. THE DEFECT IS ONE KEY. The request body posted `"audio_duration_s": 0.1`. That is the name of the FIELD (`speech_api.h:55`), not the wire key. `ParseSpeechRequest` reads `audio_duration`, which is what `docs/USAGE.md:1235` documents and what every other test uses. The key was dropped silently, `audio_duration_s` stayed at its `0.0` sentinel, and `Music3ResolveRequest` substituted `kMusic3DefaultDurationSeconds = 60.0`. So the gate asked for a sixty-second song: asked for actually ran duration 0.1 s 60 s AR frames 2 1500 denoise windows 1 8 vocoder latents 6 5167 samples/channel 3072 2645504 A ~750x job, and one that could never have passed anyway, because the case asserts the payload length 0.1 s implies. THE RECORDED DIAGNOSIS WAS WRONG, AND HOW IT WAS WRONG GENERALIZES. Four runs were placed "inside `LoadQwen3ForCausalLMWeights`" on the evidence that the four `language_model/*.safetensors` fds were open and the depth decoder's was not. Neither fact says that. `LoadBf16Direct` BORROWS the mapping when it can, and a borrowed `OwnedTensor` holds its own `shared_ptr` to it, so those fds stay open for the whole request long after the loader returned; the depth decoder's tensors are COPIED, so its `SafetensorsFile` dies and its fd closes on return. The fd pattern is a copy-versus-borrow artifact, not a program counter. What resolves it is a symbol-resolved profile. `perf record -g` on the live process, 173K samples, symbols intact because the binary was not rebuilt under it: LM weight load 0 -> 180 s 1 thread, state D, ~92 MB/s off the NAS, 3.7 s of user CPU in the whole phase AR generate 180 -> 19154 s 20 threads, LinearNoBias 42-57%, Threadpool::Barrier 25%, Bt16Avx512 12-14% acoustic 19154 s -> killed vocoder1d::ConvTranspose1d 88.5%, at 15 h 19 m Conv1d 7.7% The load is THREE MINUTES, in both binaries -- the same ~3 minutes `test_minimax_music3_llm_real` takes -- and it is I/O bound, not spinning. The `__sched_yield` frames in the earlier profile are `Threadpool::PollForWork`'s bounded hybrid poll around a genuinely compute-bound run. Two supporting measurements, taken rather than argued: the 200000 x 4096 `lm_head` scalar transpose that was suspected costs 1.388 s, and the vocoder at this request's real shape costs 5.4 s against 84 s at the capture's L=86. THE GUARD, because a silently dropped knob is what made this invisible. `audio_duration_s` is now REFUSED and named, like every other unsupported field in that function -- whose own comment already said ignoring one "returns a 200 carrying audio the caller did not ask for". Red first: the two `CHECK_THROWS` in `test_speech_api.cpp` did not throw. The guard is narrow on purpose; refusing every unknown key would break the `extra_params` bag's forward compatibility, while this one near-miss is the spelling the struct itself invites (#925). NOT CHANGED. `LinearNoBias` keeps its reduction order and its `-ffp-contract=off` scalar triple loop; W2/W3 gate it against torch. It is 42-57% of the AR profile and parallelising it over OUTPUT ROWS would not move a gated number, but that is a separate, measured change with its own evidence. No speed number is claimed: CPU only, `dgx.casa` down. GATES, all on this box, CASE and assertion counts: test_minimax_music3_e2e_real 5 | 535 (with both env vars) test_speech_api 5 | 56 test_minimax_music3_loader 21 | 1413 test_minimax_music3_ar 25 | 338 test_minimax_music3_ar_real 4 | 894 test_minimax_music3_acoustic 27 | 265 test_minimax_music3_speech 9 | 223 test_minimax_music3_quant 29 | 125 test_minimax_music3_quant_real 6 | 319 test_minimax_h3 79 | 57395 test_indextts2_family 7 | 22 test_openai_api_server 61 | 709 test_capi 63 | 617 test_qwen3_paged_engine 2 | 0 SKIP: Qwen3-0.6B/4B snapshots are dgx-only, absent here `scripts/agent-preflight.sh --no-require-role` is green except `test_cpu_x86_llamacpp_floor`, which is the known load-dependent #618 (`4 != 2`, `NO_QUIET_WINDOW` instead of `GIVING_UP`) and which the SAME tree passed 25 minutes earlier at lower load. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] Issue: #852, #925, #672
# Conflicts: # .agents/model-matrix.md # docs/FEATURES.md # docs/STATUS.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MiniMax-Music3 generates a song end to end, observed
Re-read independently of the gate at
build/music3/minimax_music3_e2e_http.wav:44100 Hz, 2 channels, 16-bit, 3072 frames, 0.069660 s, peak 156, 67 distinct
left-channel values.
The defect is one JSON key
The request body posted
"audio_duration_s": 0.1. That is the name of thefield (
speech_api.h:55), not the wire key.ParseSpeechRequestreadsaudio_duration(orduration) — whatdocs/USAGE.md:1235documents and whatevery other test in the tree uses. The key was dropped silently,
audio_duration_sstayed at its0.0sentinel, andMusic3ResolveRequestsubstituted
kMusic3DefaultDurationSeconds = 60.0.So the gate asked for a sixty-second song:
A ~750x job — and one that could never have passed anyway, because the case
asserts the payload length 0.1 s implies.
The recorded diagnosis was wrong, and how it was wrong generalizes
Four runs (85 min, 34 min, 1 h 49 m, and one of mine at 15 h 19 m) were
placed "inside
LoadQwen3ForCausalLMWeights" on the evidence that the fourlanguage_model/*.safetensorsfds were open and the depth decoder's was not.Neither fact says that.
LoadBf16Directborrows the mapping when it can(
BorrowStTensorBytes), and a borrowedOwnedTensorholds its ownshared_ptrto it — so those fds stay open for the whole request, long after theloader returned. The depth decoder's tensors are copied, so its
SafetensorsFiledies and its fd closes on return. The fd pattern is acopy-versus-borrow artifact, not a program counter.
What resolves it is a symbol-resolved profile —
perf record -gon the liveprocess, 173K samples, symbols intact because the binary was not rebuilt under
it:
D, ~92 MB/s off the NAS, 3.7 s user CPULinearNoBias42-57%,Threadpool::Barrier25%,Bt16Avx51212-14%vocoder1d::ConvTranspose1d88.5%,Conv1d7.7%The load is 3 minutes, in both binaries — the same ~3 minutes
test_minimax_music3_llm_realtakes — and it is I/O bound, not spinning. The__sched_yieldframes in the earlier profile areThreadpool::PollForWork'sbounded hybrid poll around a genuinely compute-bound run.
Two supporting measurements, taken rather than argued: the 200000 x 4096
lm_headscalar transpose that was suspected costs 1.388 s, and the vocoderat this request's real shape costs 5.4 s (against 84 s at the capture's
L=86).
The guard
A silently dropped knob is what made this invisible for four runs, so
audio_duration_sis now refused and named — like every other unsupportedfield in that function, whose own comment already said ignoring one "returns a
200 carrying audio the caller did not ask for".
RED first: the two
CHECK_THROWSintest_speech_api.cppdid not throw.The guard is narrow on purpose: refusing every unknown key would break the
extra_paramsbag's forward compatibility, while this one near-miss is thespelling the struct itself invites (#925).
Not changed
LinearNoBiaskeeps its reduction order and its-ffp-contract=offscalartriple loop — W2/W3 gate it against torch. It is 42-57% of the AR profile and
parallelising it over output rows would not move a gated number, but that is a
separate, measured change with its own evidence. No speed number is claimed:
CPU only,
dgx.casadown.Gates
All on this box (20-core x86_64, CPU only), CASE and assertion counts:
test_minimax_music3_e2e_real(both env vars)test_speech_apitest_minimax_music3_loadertest_minimax_music3_artest_minimax_music3_ar_realtest_minimax_music3_acoustictest_minimax_music3_acoustic_realtest_minimax_music3_speechtest_minimax_music3_llm_realtest_minimax_music3_quanttest_minimax_music3_quant_realtest_minimax_h3test_indextts2_familytest_openai_api_servertest_capitest_qwen3_paged_enginescripts/agent-preflight.sh --no-require-roleis green excepttest_cpu_x86_llamacpp_floor, the known load-dependent #618 (4 != 2,NO_QUIET_WINDOWinstead ofGIVING_UP), which the same tree passed 25minutes earlier at lower load.
Records
.agents/model-matrix.md,docs/FEATURES.mdanddocs/STATUS.mdconflicted onmerge and were rebuilt from main's version with only my rows reapplied; every
unrelated key was diffed against
origin/mainand is byte-identical (2 rows, 2rows and 1 row changed respectively). The tree was grepped explicitly for
conflict markers.
Issue: #852, #925, #672
🤖 Generated with Claude Code