← overview
Song models — vocals only, with and without stem separation
Eight arms × 10 genres × 3 seeds, each judged twice: as generated, and as
an isolated vocal stem. All audio 160 kbps stereo.
First: the setting that invalidated the original benchmark
dcw_enabled defaults to True in ACE-Step's
GenerationParams — a wavelet-domain correction applied at every sampler step.
Nobody set it. With it on, Whisper transcribes a 30-second XL-SFT clip as the single word
"so". With it off, the same prompt and seed transcribe as the actual lyrics.
Ruled out first, each by direct test: guidance scale (broken identically at cfg 1.0, where CFG
is effectively off), step count, the 4B LM, shift, sampler, ADG, velocity clamping, checkpoint
integrity. Only DCW moved it.
It destroys the CFG models and degrades turbo. That is why the original 4.27/5
turbo score looked plausible and went unquestioned.
Stem separation — and what it costs in bulk
The prompt asks for one voice alone, and the models mostly comply — but "mostly" hides
two different failures: bad singing, and good singing buried under a pad. Separating the vocal
first lets the rubric grade the voice rather than the arrangement.
Three separators, deliberately: separation artefacts are model-specific, so agreement across
architectures is what distinguishes a real effect from one model hallucinating.
Measured on one GH200, 240 clips each. The Roformers are
4.8× faster than Demucs here
(27× realtime against 6×). At
1.13 s per clip, separating a million 30-second clips costs about
314 GPU-hours — roughly
0.1 % of the 226,935 GPU-h the full 6,000-voice speech
corpus would cost. Separation is cheap relative to generation; it is not free.
The same rubric, on the mix and on the isolated vocal
Read score B carefully — it does not mean the same
thing in both columns.
A — singing quality & prompt match. Meaningful on both. On the stem the judge
hears the voice without accompaniment masking or flattering it. This is the comparison worth
making.
B — is it one person alone? On the mix this measures the model: did it obey
"no instruments, no second voice". On the stem it can no longer measure instruments at all
— the separator removed them by construction. It still measures extra voices, since
a vocal separator pulls voice away from instruments, not voice away from voice.
So B(mix) → B(stem) decomposes the failure: what B recovers after separation was
instrumental bleed; what stays low is a genuine second singer. B(stem) rising is not the
model getting better.
Do the three separators agree?
If a score shift were a separation artefact it would differ by architecture. BS-Roformer covers every clip; the other two cover seed 0 as a cross-check, so their n is smaller and the comparison is indicative rather than matched.
Listen — mix and isolated vocal, side by side
Seed 0 of each genre. Left is what the model produced; right is the BS-Roformer vocal stem of that exact take.
ACE-Step XL SFT — DCW on — the broken default
ACE-Step XL SFT — DCW off — fixed
ACE-Step XL turbo — DCW on
ACE-Step XL turbo — DCW off — fixed
ACE-Step 2B SFT — DCW off
ACE-Step 2B turbo — DCW off
HeartMuLa — unchanged
LeVo 2 — unchanged
Method and limits
- Judge: Gemini 3.6 Flash, same rubric and same written anchors for both columns, given
128 kbps audio — raised from the first benchmark's 48 kbps, because asking a model to
grade audio quality through a 48 kbps codec grades the codec.
- 640 judgements parsed of 640 attempted.
- The separator is not perfect. On takes that are pure noise it returns near-silence, and
the judge correctly scores that 0 — which is right, but means A(stem) for the broken arms
measures the separator finding nothing, not the model singing badly.
- Not a human listening study. One automatic judge, three seeds per genre.
- Separation throughput measured on one GH200 with these 30 s clips; longer inputs change the
per-clip figure but not the ratio between separators.
Separators: BS-Roformer (Viperx ep.317, SDR 12.98) · MelBand Roformer InstVoc
Duality V2 · Demucs v4 htdemucs_ft.