Add MiniMax Music 3 by apolinario · Pull Request #14456 · huggingface/diffusers

apolinario · 2026-08-12T15:37:05Z

Adds MiniMax Music 3 (lyrics + music description → complete songs up to 5 minutes, 44.1 kHz stereo). An 8B Qwen3 autoregressive stage predicts per-frame audio codes; its hidden states condition a 2.4B flow-matching transformer that produces Flow-VAE latents in overlapping chunks; a DAC-style decoder renders the waveform. Structurally close to Ace-Step (DiT + pipeline-local submodels) and AudioLDM2 (language model as a pipeline component).

New: MiniMaxMusic3Transformer1DModel, MiniMaxMusic3ConditionEncoder (in models/condition_embedders/), MiniMaxMusic3RVQDepthDecoder, MiniMaxMusic3Vocoder, and MiniMaxMusic3ModularPipeline (modular blocks: text encode → AR semantic generation → chunked flow-matching with guider-abstracted CFG → vocode/stitch), plus conversion script, docs, and model- and modular-pipeline tests. The scheduler is the existing FlowMatchEulerDiscreteScheduler(num_train_timesteps=1, shift=1.0, invert_sigmas=True), which reproduces the reference Euler loop exactly.

Usage:

import soundfile as sf
import torch
from diffusers import ModularPipeline

pipe = ModularPipeline.from_pretrained("MiniMaxAI/MiniMax-Music3")
pipe.load_components(dtype=torch.bfloat16)
pipe.to("cuda")

lyrics = """[verse]
Morning light filtering through the pine
Every quiet street is yours and mine
[chorus]
Softly the world begins to breathe"""

prompt = (
    "Genre: acoustic pop. BPM: 96. Key: C major. Warm and intimate, building gently into the chorus. "
    "Vocals: soft female lead, close and breathy, light stacked harmonies in the chorus. "
    "Arrangement: fingerpicked guitar and soft piano; brushed drums and upright bass enter in the chorus."
)

audio = pipe(
    prompt=prompt,
    lyrics=lyrics,
    audio_duration=60.0,
    generator=torch.Generator("cuda").manual_seed(7),
    output="audios",
)[0]

sf.write("song.wav", audio.T.float().cpu().numpy(), pipe.sampling_rate)

Parity vs the reference implementation: every converted component matches bitwise on CPU/fp32; full 30-step chunked generation matches at 57.8 dB SNR on GPU (residual comes from the fused→split QKV projections).

Weights: final home will be MiniMaxAI/MiniMax-Music3 when the model is out

Some choices I made that would be great to double check:

One generator drives the whole pipeline - the reference derives a separate seed per stage and per chunk, imo it's fine to keep the diffusers convention here.
Output is the vocoder's native 44.1 kHz; the reference server post-resamples to 32 kHz - I don't see a reason to downsample something higher quality, but let me know if following the original would be fundamental here
Generation is single-sample (prompt/lyrics are strings; the AR stage is internally batch-2 for CFG). If we want to optimize batched generation I think it may need an AR-loop redesign, which I would defer to a follow-up if someone would be interested?

HuggingFaceDocBuilderDev · 2026-08-12T15:49:50Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

…script

…I/MiniMax-Music-3

sayakpaul

Thanks!

sayakpaul · 2026-08-13T08:05:50Z

+@dataclass
+class MiniMaxMusic3TransformerOutput(BaseOutput):
+    sample: torch.Tensor


Is it not possible to use the Transformer2DModelOutput here?

Done — now returns Transformer2DModelOutput.

sayakpaul · 2026-08-13T08:06:28Z

+        return freqs.cos().contiguous(), freqs.sin().contiguous()
+
+    def forward(self, seq_len: int, device: torch.device) -> Tuple[torch.Tensor, torch.Tensor]:
+        return self._build(seq_len, device)


Consider folding into forward() so that things stay explicitly inline.

Done — folded into forward() with the cache decorator on it directly.

sayakpaul · 2026-08-13T08:07:43Z

+        self.to_k = nn.Linear(dim, self.inner_dim, bias=False)
+        self.to_v = nn.Linear(dim, self.inner_dim, bias=False)
+        self.to_out = nn.ModuleList([nn.Linear(self.inner_dim, dim, bias=False), nn.Dropout(0.0)])
+        self.set_processor(processor or MiniMaxMusic3AttnProcessor())


Suggested change

self.set_processor(processor or MiniMaxMusic3AttnProcessor())

if processor is None:

processor = self._default_processor_cls()

self.set_processor(processor)

Done (also applied to the depth decoder's attention).

sayakpaul · 2026-08-13T08:17:17Z

+        previous_latent = None
+        previous_condition = None
+        global_step = 0
+        with self.progress_bar(total=self._num_timesteps) as progress_bar:


Why not do self.num_inference_steps or put num_inference_steps * len(chunk_starts) in a separate variable and use it here? This feels like an antipattern to me.

Gone with the modular conversion — this file is removed; step accounting now lives in the loop blocks.

Per review: the conditioner moves to models/condition_embedders, the RVQ depth decoder and vocoder to models/transformers and models/autoencoders, and the standard pipeline is replaced by MiniMaxMusic3ModularPipeline (anima-style blocks, helios-style chunk loop, guider-abstracted CFG with a zeros unconditional branch). Index configs now reference all components under the diffusers library. Also applies the transformer review suggestions (Transformer2DModelOutput, inlined rotary forward, explicit processor default).

apolinario · 2026-08-13T08:40:03Z

Converted to a modular pipeline per @yiyixuxu's suggestion. Also applied Sayak's comments.

sergereview

🤗 Serge says:

Overall this is a well-structured modular integration: the block decomposition (text encoder → AR semantic generation → chunk bookkeeping → per-chunk loop with condition/prepare/set-timesteps/denoise/update sub-blocks → vocoder decode) follows the established LoopSequentialPipelineBlocks pattern, the guider is wired the same way as other modular pipelines, and I verified the chunk geometry is self-consistent (200-frame windows at ~3.445 latents/frame → 689-latent windows, ~344.5-latent hop; the 86/258 crop constants tile the song exactly and the [L-344, L-172) carry aligns with the next window's first 172 latents).

Correctness / clarity

The chunking comments in before_denoise.py, denoise.py, and decoders.py all state that neighboring windows "share 172 latent frames", but at ~3.445 latents per frame a 100-frame hop on 200-frame windows gives ~344.5 shared latent frames; 172 is only the blended/carried prefix. The code is right, the comments are wrong — worth fixing since future readers will re-derive this.
MiniMaxMusic3SemanticGenerationStep raises a misleading "the prompt ended generation immediately" error when audio_duration is shorter than one AR frame (< 0.04 s), since max_frames becomes 0.

Style / docs

MiniMaxMusic3ChunkDenoiseInner uses raw tqdm (one new bar per chunk) instead of routing through the pipeline's progress_bar/set_progress_bar_config machinery; helios has the same pattern, but a single bar or config-respecting bar would be nicer for a 14-chunk song.
Minor import-ordering slip in src/diffusers/models/__init__.py and a wrong autodoc path for Transformer2DModelOutput in the transformer docstring.

Description vs. diff

The PR description's usage example (from diffusers import MiniMaxMusic3Pipeline, .audios[0]) does not match the diff: only MiniMaxMusic3ModularPipeline / MiniMaxMusic3Blocks exist; there is no standard MiniMaxMusic3Pipeline. The in-repo docs correctly use ModularPipeline.from_pretrained, but the description should be updated to avoid confusion.

Tests

pretrained_model_name_or_path = "diffusers-internal-dev/tiny-minimax-music3" is flagged as a placeholder (TODO to move to hf-internal-testing/); please make sure the tiny repo exists and the CI tests actually run before merge. Also note the fast tests only ever exercise a single chunk (audio_duration=0.2 → 5 frames), so the multi-chunk overlap/crop path — the trickiest part of the modular loop — has no automated coverage. A small multi-chunk shape test (e.g. enough frames for 2–3 windows with tiny configs) would be valuable.

serge v0.1.0 · model: claude-fable-5 · 18 LLM turns · 33 tool calls · 1708.7s · 1520108 in / 129503 out tokens

sergereview · 2026-08-13T09:17:37Z

+
+logger = logging.get_logger(__name__)  # pylint: disable=invalid-name
+
+# Neighboring windows share 172 latent frames; the previous window's carry spans latent frames [L - 344, L - 172).


The "share 172 latent frames" claim is off: at 44100/512 ÷ 24000/960 ≈ 3.445 latents per frame, a 200-frame window is 689 latents and the 100-frame hop is ~344.5 latents, so neighboring windows share ~344 latent frames. 172 (_OVERLAP_LATENT_LENGTH) is only the carried/blended prefix of that shared span — the [L - 344, L - 172) carry is correct precisely because the shared region is ~344 long. The same "share 172" wording also appears in before_denoise.py and decoders.py; please fix all three so the (correct) constants aren't second-guessed later.

Right — the windows share ~344 latents and 172 is only the blended carry prefix. Reworded in all three files in 1a258e3.

sergereview · 2026-08-13T09:17:37Z

+            "encoder_hidden_states": (block_state.condition, torch.zeros_like(block_state.condition)),
+        }
+
+        with tqdm(total=block_state.num_inference_steps) as progress_bar:


Nit: this creates a fresh raw tqdm bar per chunk (14 bars for a 60 s song) and bypasses set_progress_bar_config. Other modular pipelines route through the pipeline-level progress_bar helper on the loop wrapper; consider driving a single bar (e.g. len(chunk_starts) * num_inference_steps) from MiniMaxMusic3ChunkLoopWrapper.__call__, or at least labeling bars with the chunk index.

Switched to a single pipeline-level progress_bar over chunks × steps, driven from the loop wrapper (1a258e3).

sergereview · 2026-08-13T09:17:37Z

+
+    @staticmethod
+    def check_inputs(block_state):
+        if block_state.audio_duration <= 0:


If audio_duration is positive but shorter than one AR frame (< 1/25 s), max_frames becomes 0, no hidden states are collected, and the user gets the misleading "the prompt ended generation immediately" error at the end of the step. Consider rejecting durations that round to zero frames here so the error names the actual cause.

Added a guard where max_frames is computed (frame rate is config-derived, so check_inputs can't see it) — 1a258e3.

sergereview · 2026-08-13T09:17:37Z

    _import_structure["transformers.dual_transformer_2d"] = ["DualTransformer2DModel"]
    _import_structure["transformers.hunyuan_transformer_2d"] = ["HunyuanDiT2DModel"]
    _import_structure["transformers.latte_transformer_3d"] = ["LatteTransformer3DModel"]
+    _import_structure["transformers.minimax_music3_rvq_depth_decoder"] = ["MiniMaxMusic3RVQDepthDecoder"]


Nit: this entry breaks the alphabetical ordering of the block — transformers.minimax_music3_rvq_depth_decoder should come after transformers.lumina_nextdit2d.

Fixed in 1a258e3.

sergereview · 2026-08-13T09:17:37Z

+                Frame-aligned conditioning from `MiniMaxMusic3ConditionEncoder`. Pass zeros for the unconditional
+                branch of classifier-free guidance.
+            return_dict (`bool`, defaults to `True`):
+                Whether to return a [`~models.transformers.transformer_minimax_music3.Transformer2DModelOutput`]


Transformer2DModelOutput is imported from models.modeling_outputs, not defined in this module, so this autodoc cross-reference won't resolve.

Suggested change

Whether to return a [`~models.transformers.transformer_minimax_music3.Transformer2DModelOutput`]

Whether to return a [`~models.modeling_outputs.Transformer2DModelOutput`]

Fixed in 1a258e3.

…e-level progress bar, zero-frame duration guard, import ordering, docstring cross-reference

github-actions Bot added size/L PR with diff > 200 LOC documentation Improvements or additions to documentation models tests utils pipelines and removed size/L PR with diff > 200 LOC labels Aug 12, 2026

apolinario force-pushed the minimax-music3-integration branch from 672f410 to 5698322 Compare August 12, 2026 15:41

github-actions Bot added the size/L PR with diff > 200 LOC label Aug 12, 2026

apolinario changed the title ~~Add MiniMax Music 3 pipeline for text-and-lyrics-to-music generation~~ Add MiniMax Music 3 Aug 12, 2026

apolinario added 7 commits August 13, 2026 07:47

Add MiniMax Music 3 transformer, pipeline sub-models, and conversion …

c19f3bc

…script

Add MiniMax Music 3 pipeline, tests, and docs

23b4ece

Wire example docstring, point diffusers-format references at MiniMaxA…

c765905

…I/MiniMax-Music-3

Use MiniMaxAI/MiniMax-Music3 as the official diffusers weights repo id

04262d7

Apply doc-builder style

820a2ef

Use dtype in examples, demonstrate the structured prompt format

4a3c539

Use soundfile in examples like the other audio pipelines

86b2bc2

sayakpaul reviewed Aug 13, 2026

View reviewed changes

apolinario force-pushed the minimax-music3-integration branch from 46564b9 to bcc6bbf Compare August 13, 2026 08:39

github-actions Bot added modular-pipelines and removed pipelines labels Aug 13, 2026

apolinario added 2 commits August 13, 2026 08:50

Point modular index loading specs at the Hub repo in the converter

8231914

Apply doc-builder style to the modular blocks docstrings

7cd51fa

sergereview Bot reviewed Aug 13, 2026

View reviewed changes

Address review-bot comments: overlap comment accuracy, single pipelin…

1a258e3

…e-level progress bar, zero-frame duration guard, import ordering, docstring cross-reference

-        self.set_processor(processor or MiniMaxMusic3AttnProcessor())
+        if processor is None:
+            processor = self._default_processor_cls()
+        self.set_processor(processor)


		logger = logging.get_logger(__name__) # pylint: disable=invalid-name

		# Neighboring windows share 172 latent frames; the previous window's carry spans latent frames [L - 344, L - 172).

	Whether to return a [`~models.transformers.transformer_minimax_music3.Transformer2DModelOutput`]
	Whether to return a [`~models.modeling_outputs.Transformer2DModelOutput`]

Uh oh!

Conversation

apolinario commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Aug 12, 2026

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

apolinario commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sergereview Bot left a comment

Choose a reason for hiding this comment

Uh oh!

sergereview Bot Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sergereview Bot Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sergereview Bot Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sergereview Bot Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sergereview Bot Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

apolinario commented Aug 12, 2026 •

edited

Loading

apolinario commented Aug 13, 2026 •

edited

Loading