← overview

MiniMax Music 3 vs LeVo 2 — one voice, no instruments

Twenty prompts spread across the measured genre space, the same prompt to both models, vocals only. Every clip judged blind, twice, by Gemini 3.6 Flash on five anchored axes, and the "no instruments" claim checked against BS-Roformer stem separation rather than left to the judge. All audio 160 kbps stereo.

Per-axis scores

Five axes, each anchored in words at every level rather than left as a bare 0-5 scale. S is the singing alone; V counts voices and ignores instruments; I counts non-vocal sound and ignores how many singers there are; G is fit to the requested style, age and gender; O is one holistic judgement. V and I are deliberately separated: the older two-axis rubric folded them together, which is exactly the confound the stem-separation row below exists to break. Each arm appears twice — as generated, and as its BS-Roformer vocal stem.

Mean over n clips ± standard deviation across clips. Each clip's score is the mean of two independent judging passes. Green ≥ 4.00, red < 3.00.

Paired, prompt by prompt

Both arms saw byte-identical prompts on the same 20 genres, so the comparison can be paired: for each prompt, MiniMax minus LeVo. Pairing removes the between-genre variance, which is much larger than the between-model variance and would otherwise bury it. A positive number favours MiniMax.

How much of this is noise

Every clip was judged twice, in two independent passes over a shuffled queue, 120 clips in both. The table is the disagreement between those two passes: it is the smallest difference this measurement can see. Any gap in the table above that is narrower than the corresponding row here is noise, and the verdict treats it as such.

The objective check: accompaniment fraction

The judge is a language model listening to audio; it can be wrong about whether a guitar is there. acc_frac is not: it is BS-Roformer's own energy split, ||instrumental||² / (||vocal||² + ||instrumental||²), the share of separated energy the separator assigned to something that is not a voice. A bare voice sits near zero; an ordinary pop mix is 0.5–0.9. The last column is the diagnostic that matters: if V (one voice alone) does not improve once the instruments are stripped out, whatever the judge heard was a second singer, because a vocal separator pulls voice away from instruments and never voice away from voice.

Method

How the 20 prompts were sampled

The frame is the 153-genre solo-voice taxonomy built earlier in this project, reduced to 50 clusters by Ward linkage on VoiceCLAP genre centroids; each cluster is represented by its medoid, a real genre rather than a synthetic average. From those 50 medoids the 20 here were chosen by farthest-point sampling on the same cosine geometry — seed the largest cluster, then repeatedly take the medoid whose minimum distance to everything already chosen is largest. No RNG, so the set is reproducible from song/scripts/mm_prompts.py.

The result is a spread rather than 20 variations of one thing: minimum pairwise cosine distance inside the 20 is 0.303, against 0.082 across all 50 medoids, so no two prompts are near-duplicates. It covers 10 of the 11 families present among the medoids, all five age bands (one child, two teen, four young adult, twelve adult, one older) and all three gender targets. Family and age coverage were checked, never imposed: they fall out of the geometry.

One prompt, given to both

Every genre has a single prompt string handed verbatim to both models, and one lyric shared across all 20 genres so that any measured difference is the singing and not the words.

A ceiling set by LeVo, applied to both. LeVo 2 feeds the description to its type_info conditioner, configured max_len: 100 and truncating with x[:, :max_len] — a silent tail cut. One token is the <|im_start|> it prepends, leaving 99. This project's standard vocals-only caption tokenises to 94–109 Qwen tokens, so 9 of these 20 prompts would have been cut, several of them losing the words “no choir or background vocals”. The caption was therefore rewritten compactly to 57–65 tokens, keeping every element (solo, a cappella, gender, age, no instruments, no backing track, no second voice, no harmony, no choir, the genre clause, dry close-mic) — and the same short caption went to both models. Shortening is prompt tailoring, so it was applied to both arms or it would not have been applied at all.

The lyric syntax differs, and has to. The words are identical; only the separators change, because the two checkpoints have incompatible input contracts:

• MiniMax Music 3 — each [tag] on its own line. Text sharing a line with a leading tag is dropped by the checkpoint (diffusers/modular_pipelines/minimax_music3/encoders.py::_normalize_lyrics).
• LeVo 2 — one line, ' ; ' between sections, '.' between phrases.

The one lyric, in both forms, in full:

MiniMax Music 3
[verse]
I left the window open half the night
counting all the reasons I should stay
the morning came in quiet as a hand
and took the argument away
[chorus]
So carry me, carry me home
over the water and back through the town
carry me, carry me home
before the light goes down

LeVo 2
[verse] I left the window open half the night. counting all the reasons I should stay. the morning came in quiet as a hand. and took the argument away. ; [chorus] So carry me, carry me home. over the water and back through the town. carry me, carry me home. before the light goes down.

What was reused and what was generated

Nothing was reused. All 60 clips were generated for this comparison. The existing LeVo 2 corpora were checked rather than assumed: all 20 genres appear in TTS-AGI/levo2-vocals-captioned (8,582 clips, 85 genres) and 15 of 20 in levo2-vocals-gender-rebalance, but those corpora use short tag-style descriptions (“male, rock, sixties surf pop lead vocal, …”) — 210 and 60 distinct prompts respectively, and zero of them match any of the 20 prompts used here. Under a rule of identical prompts to both models, genre-matched is not prompt-matched, so none of them were eligible.

Generation

• MiniMax Music 3 — the diffusers modular pipeline. The model card's three runtimes are SGLang-Omni, diffusers and ComfyUI; the sglang-omni build present on this system has no minimax_music3 model, so diffusers was the route taken. It lives in an unmerged PR, installed with --no-deps into a directory on PYTHONPATH so no existing environment was modified. One GH200 per shard, four shards, bf16, peak 22.8 GiB — it fits the 24 GB card the model card claims. 61 s of wall clock per clip (20 GPU-minutes for all 20), 44.1 kHz stereo, audio_duration=45, 30 flow-matching steps per chunk. The language model emits an end-of-audio token when it has finished the lyric, so the renders run 15–30 s (mean 22.4 s) rather than the full 45.
• LeVo 2 / SongGeneration-v2-large — gen_type='vocal' for the main arm and gen_type='mixed' for the control, otherwise identical settings (temperature 0.9, top-k 50, cfg 2.0, 30 s). 50 s per clip, 48 kHz stereo, renders 20–30 s (mean 28.2 s). The two arms therefore differ in mean clip length by about 6 s, which is stated because a shorter clip gives an instrument less time to appear; the objective accompaniment fraction is an energy ratio and is not affected by it.
• Both arms: one seed per prompt, the same seed (4242) for every clip, 30 s kept.

What it took to run MiniMax Music 3 here

Recorded because “it runs” was not a given: aarch64 GH200, torch already installed, compute nodes with no internet. Nothing was pip installed without --no-deps, and no existing environment was modified.

• Weights — 57 GB in the repo; only the 28 GB the diffusers path needs was fetched (the 9.8 GB flowmatching_vae.pth and the 48-shard qwen_7B/ tree serve the SGLang runtime). Fetched on the login node, 32 files in 29 s.
• diffusers — MiniMax Music 3 is in an unmerged PR (huggingface/diffusers@dafe3733). Installed with uv pip install --no-deps --target song/dfmm and put on PYTHONPATH, so env_song's own diffusers 0.39.0 is untouched. That is the only thing installed for this work.
• The hub id does not work offline. modular_model_index.json names the repo id for every component, so with a warm cache and HF_HUB_OFFLINE=1 the loader still calls the Hub API and fails. Fixed by materialising a directory of symlinks with the component paths rewritten to point at it (mm_localise.py).
• The tokenizer folder is incomplete for the class the pipeline declares. The pipeline asks for the slow Qwen2Tokenizer, which wants vocab.json + merges.txt; tokenizer/ ships only tokenizer.json, and the protobuf fallback is not installed. The repo's other tokenizer copy has both files, and it was checked to be the same tokenizer — identical vocab, merges and added tokens, differing only in serialised bytes — before being linked in.
• One import from the future. diffusers/pipelines/pipeline_utils.py imports get_cached_repo_tree / CachedRepoTreeNotFoundError, which postdate huggingface_hub 0.36.2. Shimmed in-process to raise the "nothing cached" error rather than upgrading huggingface_hub underneath ACE-Step.
• It then just works. bf16, peak 22.8 GiB, no flash-attention wheel needed, no custom kernels, 60 s per clip on one GH200. The vocoder reports 44.1 kHz, not the 32 kHz the model card states.

Verifying the LeVo 2 baseline rather than assuming it

The prior figures for LeVo 2 on this task were singing quality 4.40, one-voice compliance 4.07, no change across stem separation, and a measured accompaniment fraction of 0.0037. Two of those reproduce and two do not, for reasons worth stating:

• No change across stem separation: reproduced exactly. LeVo's one-voice score is 4.28 on the mix and 4.28 on the stem — a delta of +0.00. There is nothing for a separator to take away.
• Accompaniment fraction: lower still, 0.0000 (max over 20 clips 0.0002). Measured here as the separator's own energy split rather than by the earlier method, so the two numbers are not the same statistic; both say the same thing.
• Singing quality 3.77, not 4.40, and one-voice 4.28, not 4.07. Neither is a contradiction: this is a different rubric with different anchors, a different prompt, one seed per genre, and a genre set chosen for spread rather than for what LeVo does well. The prior numbers are not a baseline this run can be scored against, which is why they are not used as one anywhere above.

Judging

Gemini 3.6 Flash over the native generateContent endpoint with thinkingConfig: {"thinkingLevel": "low"} — {"thinkingBudget": 0} returns HTTP 400 for this model on this endpoint, verified against the live API before the run. Audio at 128 kbps, because one of the axes is audio quality and the codec must not be the thing being graded.

Blind by construction. Each request carries exactly one clip plus the prompt and lyrics it was generated from. No model name, no file name, no arm label, and never two clips in one request, so there is nothing in the request that could identify the generator; the queue is shuffled as well, so dispatch order carries no signal either. Each clip was judged twice, in two independent passes with separate caches.

All 20, side by side

Each row shows the prompt exactly as both models received it, both takes, the five per-clip scores (mean of the two judging passes), the accompaniment fraction, and the judge's own one-line description of what it heard. The note under each row is generated from the numbers, not written by hand.

a cappella solo voice, in their thirties; unaccompanied, no instruments, no backing track, no second voice, no harmony, no choir; sami joik, cyclical wordless vocable chant, guttural and hypnotic; dry close-mic, no reverb

a cappella solo voice, female, a teenager; unaccompanied, no instruments, no backing track, no second voice, no harmony, no choir; teen pop, bright youthful thin-bodied voice, eager and unweathered; dry close-mic, no reverb

a cappella solo voice, male, in their twenties; unaccompanied, no instruments, no backing track, no second voice, no harmony, no choir; punk, shouted snarling barked delivery, aggressive and untrained; dry close-mic, no reverb

a cappella solo voice, in their twenties; unaccompanied, no instruments, no backing track, no second voice, no harmony, no choir; indie pop, slightly nasal untrained charm, close-mic breathiness, casual; dry close-mic, no reverb

a cappella solo voice, male, in their twenties; unaccompanied, no instruments, no backing track, no second voice, no harmony, no choir; afrobeats, lilting melodic pidgin-inflected phrasing, relaxed buoyant swing; dry close-mic, no reverb

a cappella solo voice, female, in their thirties; unaccompanied, no instruments, no backing track, no second voice, no harmony, no choir; gothic symphonic metal soprano, operatic vibrato over heaviness, ethereal; dry close-mic, no reverb

a cappella solo voice, male, a child of about eight; unaccompanied, no instruments, no backing track, no second voice, no harmony, no choir; solo chorister boy, pure vibrato-free treble, echoing cathedral clarity; dry close-mic, no reverb

a cappella solo voice, male, in their thirties; unaccompanied, no instruments, no backing track, no second voice, no harmony, no choir; power metal, soaring operatic high tenor, sustained ringing screams; dry close-mic, no reverb

a cappella solo voice, male, a teenager; unaccompanied, no instruments, no backing track, no second voice, no harmony, no choir; boy-band lead, light high tenor, earnest and smooth, adolescent brightness; dry close-mic, no reverb

a cappella solo voice, male, in their thirties; unaccompanied, no instruments, no backing track, no second voice, no harmony, no choir; comic patter song, extremely fast precise consonants, breathless wit; dry close-mic, no reverb

a cappella solo voice, in their thirties; unaccompanied, no instruments, no backing track, no second voice, no harmony, no choir; bel canto, effortless even legato across registers, elegant messa di voce swells; dry close-mic, no reverb

a cappella solo voice, in their thirties; unaccompanied, no instruments, no backing track, no second voice, no harmony, no choir; cabaret, arch knowing delivery, spoken asides sliding into sung line; dry close-mic, no reverb

a cappella solo voice, male, in their thirties; unaccompanied, no instruments, no backing track, no second voice, no harmony, no choir; forro, warm northeastern brazilian accent, rustic bouncing delivery; dry close-mic, no reverb

a cappella solo voice, female, in their thirties; unaccompanied, no instruments, no backing track, no second voice, no harmony, no choir; spanish saeta, unaccompanied processional cry from a balcony, raw and piercing; dry close-mic, no reverb

a cappella solo voice, male, in their thirties; unaccompanied, no instruments, no backing track, no second voice, no harmony, no choir; mexican ranchera, huge open-throated grito, proud sustained belt; dry close-mic, no reverb

a cappella solo voice, male, in their thirties; unaccompanied, no instruments, no backing track, no second voice, no harmony, no choir; quiet storm R&B, smooth low seductive croon, silky sustained falsetto; dry close-mic, no reverb

a cappella solo voice, male, over sixty; unaccompanied, no instruments, no backing track, no second voice, no harmony, no choir; delta blues, rough weathered voice, bent notes, moaning field-holler inflection; dry close-mic, no reverb

a cappella solo voice, male, in their thirties; unaccompanied, no instruments, no backing track, no second voice, no harmony, no choir; dancehall, rapid patois toasting sliding into melody, rhythmic and percussive; dry close-mic, no reverb

a cappella solo voice, male, in their thirties; unaccompanied, no instruments, no backing track, no second voice, no harmony, no choir; pure falsetto, light hooty unsupported head voice, fragile; dry close-mic, no reverb

a cappella solo voice, male, in their twenties; unaccompanied, no instruments, no backing track, no second voice, no harmony, no choir; britpop, adenoidal regional-accented drawl, cocky flat delivery; dry close-mic, no reverb

Built 2026-08-14. 60 clips generated, 60 judged twice on the mix and 60 on the vocal stem. Scripts: song/scripts/mm_prompts.py, gen_minimax.py, gen_levo.py, mm_stems.py, mm_judge.py, mm_agg.py, mm_page.py.