LeVo 2 throughput

The slowest model in the benchmark was running at batch 1 because its inference code asserts batch 1, and spending 94 % of its time in a decode loop that was CPU‑dispatch bound, not GPU bound. Fixing both gives 13.6× realtime on one GH200, measured end to end.

0.67×baseline realtime factor, agg.json

13.6×best measured, 30 s clips, batch 32

8.5×best measured, full 2.7 min songs, batch 8

−68,400GPU‑hours saved per 1 M 3‑minute songs

the finding architecture phase profile batch sweep optimisations best config & cost what didn't work what's next sources

The finding, in one paragraph

LeVo's inference code is hard‑wired to batch 1 — CodecLM._prepare_tokens_and_attributes opens with assert len(lyrics) == 1, and LmModel.generate computes num_samples from the input then throws it away, calling prepare_condition_tensors(batch_size=1, ...). Our driver gen_levo.py was correctly calling the only API that exists. Separately, a torch profiler run shows each decode step issues ~2,437 individual op dispatches for 9 ms of actual GPU work, while wall clock per step is 43 ms. The loop is bound by CPU dispatch, not by the GPU. Those two facts compound: because the CPU cost is per step and not per batch element, batching is very nearly free, and capturing the step in a CUDA graph removes most of what remains.

What the model actually is

Read from songgeneration_v2_large/config.yaml and the loaded checkpoint, not from the paper. LeVo 2 large is two stacked LLaMA decoders plus a flow‑matching vocoder:

component	shape	params	role
`transformer` (main)	36 layers, d=2048, ffn=11008, 16 heads, no GQA	3.106 B	predicts codebook 0 (the "mixed" stream)
`transformer2` (sub)	12 layers, same width	1.080 B	takes main hidden state + codebook 1/2 embeddings, drives two linear heads for vocal & bgm
conditioner embeddings	2 × Qwen2 vocab (151,652 × 2048)	~0.94 B	lyrics + tag text
LM total	fp16	5.122 B	~10.2 GiB of weights
Flow1dVAE vocoder	non‑causal GPT‑2, 50 Euler steps, CFG 1.5	separate module	codes → 48 kHz stereo

Three things about the decode loop matter for throughput and are easy to miss:

Both transformers run every step — 48 layers per token, not 36.
CFG doubles the batch internally. _sample_next_token does torch.cat([sequence, sequence], dim=0), so a user batch of 32 is a model batch of 64.
The delay pattern costs a flat 250 steps. delays: [0, 250, 250] at 25 Hz is a 10‑second offset between the mixed stream and the vocal/bgm streams. A 30 s clip needs 750 + 250 = 1,000 steps — 25 % pure overhead. A 3 minute song needs 4,500 + 250 = 4,750 — only 5 %. Longer songs are intrinsically more efficient per second of audio.

The conditioning prefix is a fixed 952 tokens (lyrics 600 + tags 100 + audio‑prompt slot 252), padded to constant length regardless of content. That is what makes batching straightforward: every sequence in a batch has an identical‑length prefix, so no ragged padding or attention‑mask work is needed.

Where the 44 seconds go

Batch 1, 30 s target, dedicated compute node, torch.cuda.synchronize() around every phase.

phase	seconds	share	note
LM token generation	34.26	94.0 %	769 steps @ 22.45 steps/s = 44.5 ms/step
Vocoder — CFM Euler solve	1.50	4.1 %	50 steps, CFG 1.5
Vocoder — VAE decode	0.69	1.9 %	latents → 48 kHz stereo
total	36.45		20.72 s audio → xrt 0.568 (model EOS'd early)
model load (once)	145.5	—	LM 126.8 s + vocoder 18.7 s; amortised over a run

Peak VRAM at batch 1: 16.78 GiB of 95. Roughly 78 GiB per GPU was sitting idle.

Why the LM step costs 44 ms

measurement	value	what it says
wall clock per decode step	43.5 ms
GPU time per step (profiler)	9.0 ms	the GPU is idle ~79 % of the step
CPU time per step (profiler)	58.4 ms	dispatch, not compute, is the critical path
op dispatches per step	~2,437	~24 µs of CPU per op
of which `aten::mm`	340/step, 41 % of GPU time	the real work
of which `aten::cat`	195/step, 21 % of GPU time	the KV cache being reallocated and copied every step
per‑layer: attention	0.656 ms	eager `matmul→softmax(fp32)→matmul`
per‑layer: MLP	0.108 ms	4× the FLOPs of attention, 6× less time

The MLP is where the FLOPs are and it takes a sixth of the time attention does. That is the signature of a latency‑bound loop. Two structural causes: attention runs the eager path (flash_attn has no aarch64 wheel, so the vendored LlamaAttention falls back to explicit matmul + fp32 softmax), and the KV cache grows by torch.cat on every layer on every step, reallocating and recopying the whole cache 96 times per token.

Batch‑size sweep

Stock kernels, batching patched in, vocoder run as an independent stage at batch 4. Dedicated compute node, one GH200. PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True.

batch	wall (s)	audio (s)	× realtime	s / song	LM (s)	LM steps/s	CFM (s)	VAE (s)	peak VRAM
1	36.5	20.7	0.57	36.5	34.3	22.45	1.5	0.7	16.8 GiB
2	37.6	41.8	1.11	18.8	34.3	23.85	2.4	0.9	19.4 GiB
4	45.9	90.6	1.97	11.5	39.3	24.55	4.8	1.7	24.8 GiB
8	55.1	186.0	3.38	6.9	41.9	23.87	9.6	3.5	35.0 GiB
16	83.3	375.8	4.51	5.2	56.0	17.84	20.1	7.0	54.8 GiB
24	117.2	585.8	5.00	4.9	76.3	13.11	30.2	10.4	75.2 GiB
32	CUDA OOM — 95 GiB exhausted by the `torch.cat` KV cache

Batch 1→8 is almost perfectly linear: LM steps/s is flat (22.45→23.87 — it actually goes up), so 8× the batch is 5.9× the throughput, with the shortfall entirely in the vocoder, which does not benefit from batching (it is already compute‑saturated; CFM time scales linearly with batch). Beyond 8 the LM starts to slow as it becomes genuinely GPU‑bound, and beyond 24 the stock cache runs out of memory.

Two correctness bugs surface only at batch > 1, both fixed in the harness:
1. Batch‑wide truncation. CodecLM.generate does length = torch.nonzero(tokens == eos)[:, -1].min() — the minimum EOS position across the whole batch — then truncates everything to it. Correct at batch 1; at batch 16 it silently cuts every song down to the length of the shortest one. We keep per‑row lengths and trim each song's audio individually.
2. Batch‑1 assumptions in sampling. The repetition‑penalty pool is built with next_token.squeeze(), which collapses the batch dimension, and the token ban is applied as logits[0][0][...] — row 0 only. Both were rewritten to be per‑row.

Outputs are not reproducible across batch sizes. With an identical seed and identical lyrics, row 0's generated length was 20.72 s at batch 1, 19.16 s at batch 2, 17.24 s at batch 4 and 16.28 s at batch 16. Sampling consumes one shared RNG stream in a different order as batch changes, and batched GEMM reductions associate differently. A seed reproduces a song only at a fixed batch size. Plan around this if any provenance or regeneration guarantee is needed.

Optimisations, measured one at a time

Each row changes exactly one thing relative to the row above it, so the gains are attributable.

change	batch	LM ms/step	gain on LM	end‑to‑end ×RT	VRAM	where measured
stock, batch 1	1	44.5	—	0.57	16.8	node
+ batching (the whole answer)	8	41.9	6.3× aggregate	3.38	35.0	node
+ torch SDPA instead of eager attention	8	35.4	1.18×	3.99	34.7	node
+ CFM 50 → 10 Euler steps	8	unchanged	vocoder 9.6→2.0 s	4.28	35.3	node
+ static KV cache (no `cat`)	8	35.6	~flat, but unlocks graphs & batch 32+	—	23.2	login
+ CUDA graph on the decode step	8	13.9	2.56×	7.04	32.6	login
+ batch 16, CFM 10	16	21.7	—	11.72	50.7	login
+ batch 32, CFM 10	32	34.0	—	13.56	86.8	login

Static KV cache + CUDA graphs — the second lever

Stock LeVo cannot be graphed at all: LlamaAttention.forward grows the cache with torch.cat, so every tensor shape changes every step. Replacing it with a preallocated buffer written by index_copy_ at a position index makes every shape static, at which point the whole 48‑layer step captures into a single torch.cuda.CUDAGraph. LM decode only:

batch	static cache, eager (ms/step)	CUDA graph (ms/step)	graph speedup	LM‑only ×realtime	peak VRAM
8	35.6	13.9	2.56×	23.1	23.2 GiB
16	36.0	19.5	1.85×	32.9	36.6 GiB
32	37.6	30.9	1.22×	41.5	63.5 GiB
48	46.1	42.0	1.10×	45.7	90.3 GiB

The graph speedup shrinks as batch grows — 2.56× at batch 8 down to 1.10× at batch 48 — which is exactly what the profile predicts. CUDA graphs remove CPU dispatch overhead; once batch is large enough that the GPU work exceeds that overhead, there is nothing left to hide. Batching and CUDA graphs attack the same bottleneck, so they do not multiply. Anyone stacking both should expect the combined figure, not the product.

The fastest configuration that actually works

Recommended: batch 16, static KV cache, CUDA‑graphed decode step, torch SDPA, vocoder as a separate stage at batch 8 with 10 CFM steps.
Measured 11.7× realtime at 50.7 GiB for 30 s clips — leaving comfortable headroom on a 95 GiB card. Batch 32 reaches 13.6× but at 86.8 GiB, which is too close to OOM to run unattended across thousands of jobs. All outputs verified as real audio (see below).

config	batch	wall (s)	audio (s)	×RT	s/song	LM (s)	vocoder (s)	VRAM
baseline (`agg.json`, production)	1	44.4	29.8	0.673	44.4	~42	~2.3	16.8
graph + batch, 30 s clips	8	29.0	204.0	7.04	3.6	15.8	13.2	32.6
graph + batch + CFM 10, 30 s clips	16	33.4	391.3	11.72	2.1	21.7	11.7	50.7
same, pushed	32	59.8	810.5	13.56	1.9	34.0	25.8	86.8
full songs (2.7 min each), CFM 10	8	152.9	1302.5	8.52	19.1	120.4	32.5	54.9

Cost for 1 million 3-minute songs

The full‑song row is the one to cost against. It is the only measurement made on genuinely multi‑minute output. Song length is driven by the lyrics, not by --duration — asking for 180 s with a single [verse] still produces ~27 s, because the model emits EOS when the lyrics are done. Full structured lyrics ([intro-long] ; [verse] ; [chorus] ; [inst-long] ; [verse] ; [chorus] ; [outro-long]) produced 163 s per song, which is what the 8.52× figure is measured on. The 3‑minute numbers below scale that by 180/163 — that scaling is an extrapolation; everything else in this table is measured.

configuration	×realtime	GPU‑s per 3‑min song	GPU‑hours per 1 M	node‑hours (4 GPU)	core‑hours billed
baseline, batch 1	0.673	267.5	74,300	18,580	5.35 M
batching only, batch 24, stock kernels	5.00	36.0	10,000	2,500	720 k
recommended (graph + batch + CFM 10)	8.52	21.1	5,870	1,470	423 k

Saving versus the current pipeline: about 68,400 GPU-hours, or 4.9 M billed core-hours, per million songs — a 12.7× reduction. For reference, ACE‑Step XL turbo currently sits at 2.82× realtime; the recommended LeVo config is 3.0× faster than that, so LeVo would go from being the pipeline's bottleneck to being its fastest arm.

Output validation

A fast wrong answer is worthless, so every batched configuration was checked for real audio rather than silence or truncation. Frames above −40 dBFS, on 50 ms windows:

file	channels	duration	RMS	peak	active frames
batch 16, CUDA graph	2	30.0 s	0.136	1.22	79.5 %
batch 16, CUDA graph	2	30.0 s	0.138	0.92	65.3 %
batch 32, CUDA graph	2	30.0 s	0.073	1.16	74.8 %
batch 32, CUDA graph	2	30.0 s	0.137	0.98	61.5 %
production batch‑1 reference	2	30.0 s	0.237	1.21	92.5 %
production batch‑1 reference	2	30.0 s	0.176	1.03	83.5 %

Batched outputs are unambiguously real vocal audio at full length. They are somewhat quieter and sparser than the production reference, which is consistent with the different (shorter, single‑verse) test lyrics used here rather than with a batching artefact — but this has not been confirmed by a listening test or by the judge rubric, and should be before a large run. The RMS spread within a single batch (0.07–0.14 at batch 32) is normal variation across prompts.

What did not work, and why

FlashAttention 2 — unavailable on aarch64. Confirmed empirically: flash_attn is absent from the env, and upstream ships essentially no aarch64 wheels (one asset in the whole v2.8.3 release, for cp312/cu13). The config's use_flash_attn_2: true is dead on this machine; the vendored attention silently falls back to the eager path. torch SDPA is the correct substitute and does work — its flash and mem‑efficient backends are compiled into libtorch and gated on the GPU backend, not the host ISA (verified: both report enabled on this box). It bought 1.18× on the LM.
xformers — does not exist for aarch64. No wheel on PyPI or download.pytorch.org, and source builds fail on ARM NEON headers. Do not plan around it.
Batch 32 with the stock KV cache — OOM. Exhausts all 95 GiB, because torch.cat transiently doubles the cache on every layer of every step. The static cache is not merely an optimisation here; it is what makes batch 32 reachable at all.
Weight‑only quantisation (AWQ/GPTQ/INT4) — not attempted, and should not be. Two independent reasons. The measured profile says we are dispatch‑bound and then compute‑bound, never weight‑bandwidth‑bound at batch > 8, so there is no bandwidth to save. And the published evidence agrees: on H100‑class bandwidth, quantisation adds essentially nothing over compilation (130.5 vs 129.3 tok/s at batch 1; 83.6 vs 83.9 at batch 32), while naive INT4 RTN on audio LMs is documented to destroy end‑of‑audio prediction outright. Also: AutoAWQ and AutoGPTQ are both archived and x86‑only.
torch.compile — not attempted; manual CUDA graphs were the shorter path. LeVo vendors its own LLaMA and asserts transformers < 4.40, using legacy tuple KV caches. Compiling it would have required writing the static cache first regardless — and once the static cache exists, torch.cuda.CUDAGraph captures the step directly with no dynamo breakage to debug. This is a deliberate choice, not a gap: it is possible torch.compile with kernel fusion would beat a raw graph, and that is untested.
vLLM — not viable here as an off‑the‑shelf drop‑in. Stock vLLM's sampler is one token per sequence per step; LeVo needs three codebooks sampled in parallel plus CFG. Notably, Tencent's own v2 Space does run LeVo on vLLM — but only via four patched vLLM source files that add a guidance_scale field to SamplingParams and apply CFG inside the sampler, and it still submits one song per request with enforce_eager=True. An in‑progress vLLM‑Omni PR for LeVo 2 exists and explicitly pins max_num_seqs: 1. So the upstream path offers no batching today and would need the same work we did here.
Asking for longer songs via --duration — does nothing on its own. See the note above: length comes from the lyrics.
Login‑node measurements are unreliable for this workload, which is worth recording as a methodology finding: because the loop is CPU‑bound, LM timings on a shared login node varied ±30 % run to run (58.5 / 78.0 / 66.6 s for the same batch‑16 config). All stock‑kernel and SDPA numbers here come from a dedicated node; the CUDA‑graph numbers were taken on the login node and are, if anything, conservative.

Ranked next steps, not implemented

#	optimisation	expected gain	effort	why
1	Bucket the static KV buffer to the actual sequence length	1.15–1.3× on long songs	low	Our decode attends over the full `maxlen` buffer, not the filled prefix. At maxlen 5800 the step cost rose 21.7→25.3 ms for the same batch. Round the buffer up to the actual song length instead of a global maximum.
2	Run the vocoder on a different GPU from the LM	1.3–1.5× end‑to‑end	medium	At batch 32 the vocoder is 43 % of wall clock and runs strictly after the LM. The two stages have opposite profiles (LM latency‑bound, vocoder compute‑bound), so pipelining them across 2 of the 4 GPUs should overlap almost completely. This is exactly the pattern the TTS serving stacks converged on.
3	Drop `transformer2`'s unused `lm_head`	~2 %	trivial	A 2048×16385 matmul plus an fp32 cast is computed every step and the result is discarded — only `hidden_states` is used. One line.
4	Graph the sampling tail as well	5–10 %	low	Softmax, top‑k and the repetition‑penalty scatter still run eagerly outside the graph, ~100 dispatches/step.
5	Continuous batching / rolling admission	1.2–1.4×	high	A fixed batch runs until the last song emits EOS. Measured spread within one batch was 16–30 s, so ~25 % of decode slots are wasted on finished rows. Admitting a new song when one finishes recovers that.
6	Generate longer songs per call	1.2× on 30 s workloads	trivial	The 250‑step delay is a flat cost: 25 % overhead on a 30 s clip, 5 % on a 3‑minute one. The vocoder also pads anything shorter than 40 s up to a 40 s window. If the product allows longer outputs, they are strictly cheaper per second.
7	FP8 / W8A8 on the LM	1.1–1.4× at large batch	high	The one quantisation family with evidence of helping in the compute‑bound regime (~40 % at batch 256 vs ~10 % for W4A16). Needs Transformer Engine and careful audio‑quality validation.
8	Reduce CFG to 1.0	up to 2×	trivial — but changes output	CFG doubles the model batch for free quality. Turning it off halves LM compute outright. This is a quality decision, not an engineering one, and would need a judge‑rubric A/B before adoption.

One warning worth carrying into any large run. Two independent audio‑serving projects report catastrophic output‑quality outliers specifically at large CUDA‑graph batch sizes (SGLang‑Omni measured 99–100 % word error at graph batch ≥16 against a 0.1–0.4 % baseline, worse with torch.compile also enabled). Our own outputs look correct by RMS and activity, but that is a coarse check. Before committing thousands of GPU‑hours, run the existing judge rubric on a few hundred batch‑16 and batch‑32 clips and compare against the batch‑1 scores.

Sources

Measured here

song/scripts/bench_levo.py — batching patches, phase instrumentation, batch sweep
song/scripts/levo_cudagraph.py — static KV cache, CUDA graph capture, end‑to‑end generation
song/scripts/bench_levo.sbatch — the dedicated‑node sweep; results in song/bench/node.jsonl
song/bench/wav_cg*/ — audio from the batched configurations

LeVo / SongGeneration

The official tencent-ailab/SongGeneration repo now returns 404 and is absent from the org's repo list; the last Wayback snapshot is 2026‑03‑30. Its issue tracker is therefore unavailable. Archived README — states RTF 0.82 on H20 for v2‑large, close to our 0.67 on GH200.
The official HF Space was renamed, so its git history is the surviving primary source: huggingface.co/spaces/tencent/SongPrep — contains the v2 vLLM path, the patched vllm_hacked/ sampler, and num_steps=10 on the vocoder.
aiaiai666/SongGeneration2-WebUI — the only public copy of Tencent's vLLM patches; also reports replacing the vocoder's eager GPT‑2 attention with SDPA cutting that block's peak memory 250.7 → 26.7 MiB.
vllm-omni PR #3937 (open, max_num_seqs: 1) and issue #3390 — the in‑progress LeVo 2 vLLM port.
AWS Neuron PR #108 (merged) — the only other published LeVo batching number: batch 1→2 gives 1.44× throughput on Trainium2.
LeVo 2 paper (arXiv 2606.30642) — contains no RTF, parameter or step‑count figures.

Technique and aarch64 availability

Parler-TTS INFERENCE.md — closest published analogue: SDPA 1.4×, torch.compile default 1.4×, reduce-overhead (CUDA graphs) 3–4×, requires a static cache. Matches our CUDA‑graph result in kind and magnitude.
PyTorch on aarch64 (May 2026) and pytorch#146406 — SDPA flash/mem‑efficient backends are gated on GPU backend, not host ISA; they work on GH200.
flash-attention #1866 (GH200) — open, unanswered; no aarch64 wheels.
xformers #1142 — aarch64 source builds fail on NEON headers; unfixed.
TheStageAI Elastic-musicgen-large — batch‑resolved tok/s for a comparable music LM, and the evidence that on H100‑class bandwidth compilation is the whole gain and quantisation adds ~nothing.
SqueezeBits weight‑only quantisation — ~2× at batch 1 collapsing to ~1.15× at batch 256.
voxtral-int4 — INT4 RTN on an audio LM ran faster and produced infinite gibberish; the end‑of‑audio failure mode.
sglang-omni #478 — catastrophic WER outliers at CUDA‑graph batch ≥16; the reason for the validation warning above.
Fish Speech S2 Pro recipe and sglang-omni PR #574 — stage‑separating the AR LM from the codec decoder onto different GPUs, and batching the vocoder independently (+7.8 % throughput, −20 % p99).
vLLM logits processors design — the one‑token‑per‑sequence sampler contract that blocks multi‑codebook models.

All timings on NVIDIA GH200 (95 GiB), aarch64, torch 2.7.1+cu128, transformers 4.37.2, songgeneration_v2_large, gen_type=vocal, temperature 0.9, top‑k 50, CFG 1.5. Rows marked "node" were measured on a dedicated compute node; rows marked "login" on the login‑node GH200 and are conservative.