Measured throughput for Gemma-4-E4B on this cluster, extrapolated to the full corpus
Gemma-4-E4B-it on one GH200, plain transformers, batched, generating lines of the
real prompt shape and length (30–50 words, ~110 new tokens):
| batch | tokens/s | lines/s | peak VRAM |
|---|---|---|---|
| 32 | 379 | 6.8 | 16.7 GB |
| 64 | 663 | 11.6 | 17.5 GB |
| 128 | 1299 | 22.8 | 18.6 GB |
Scaling is close to linear across this range and VRAM barely moves (16.7 → 18.6 GB), so batch 128 is not near any limit — larger batches would very likely go further. Batch 128 is used as the conservative baseline below.
A voice needs 125 distinct texts, not 832: the four intensity/containment conditions of one emotion deliberately share a sentence, and English and German are translations of the same line. So it is 125 slots × 2 languages = 250 lines per text set.
The open choice is how many distinct sets to make. One set shared by all 6,000 voices is cheapest but means every voice says the same words; a unique set per voice is the richest corpus.
| scale | model | engine | GPU-h | core-h | % REFORMO left |
|---|---|---|---|---|---|
| one shared text set for all voices 250 lines | Gemma-4-E4B-it (MEASURED) | transformers, batch 128 (MEASURED) | 0.0 | 0 | 0.000 % |
| Gemma-4-E4B-it (MEASURED) | vLLM (assumed 6×) | 0.0 | 0 | 0.000 % | |
| Gemma-4-E4B-it (MEASURED) | SGLang (assumed 8×) | 0.0 | 0 | 0.000 % | |
| Gemma-3-12B bf16 (assumed ⅓ speed) | transformers, batch 128 (MEASURED) | 0.0 | 1 | 0.000 % | |
| Gemma-3-12B bf16 (assumed ⅓ speed) | vLLM (assumed 6×) | 0.0 | 0 | 0.000 % | |
| Gemma-3-12B bf16 (assumed ⅓ speed) | SGLang (assumed 8×) | 0.0 | 0 | 0.000 % | |
| Gemma-3-12B 4-bit quantised (assumed 0.6×) | transformers, batch 128 (MEASURED) | 0.0 | 0 | 0.000 % | |
| Gemma-3-12B 4-bit quantised (assumed 0.6×) | vLLM (assumed 6×) | 0.0 | 0 | 0.000 % | |
| Gemma-3-12B 4-bit quantised (assumed 0.6×) | SGLang (assumed 8×) | 0.0 | 0 | 0.000 % | |
| 100 distinct sets 25,000 lines | Gemma-4-E4B-it (MEASURED) | transformers, batch 128 (MEASURED) | 0.3 | 22 | 0.000 % |
| Gemma-4-E4B-it (MEASURED) | vLLM (assumed 6×) | 0.1 | 4 | 0.000 % | |
| Gemma-4-E4B-it (MEASURED) | SGLang (assumed 8×) | 0.0 | 3 | 0.000 % | |
| Gemma-3-12B bf16 (assumed ⅓ speed) | transformers, batch 128 (MEASURED) | 0.9 | 66 | 0.000 % | |
| Gemma-3-12B bf16 (assumed ⅓ speed) | vLLM (assumed 6×) | 0.2 | 11 | 0.000 % | |
| Gemma-3-12B bf16 (assumed ⅓ speed) | SGLang (assumed 8×) | 0.1 | 8 | 0.000 % | |
| Gemma-3-12B 4-bit quantised (assumed 0.6×) | transformers, batch 128 (MEASURED) | 0.5 | 37 | 0.000 % | |
| Gemma-3-12B 4-bit quantised (assumed 0.6×) | vLLM (assumed 6×) | 0.1 | 6 | 0.000 % | |
| Gemma-3-12B 4-bit quantised (assumed 0.6×) | SGLang (assumed 8×) | 0.1 | 5 | 0.000 % | |
| 1,000 distinct sets 250,000 lines | Gemma-4-E4B-it (MEASURED) | transformers, batch 128 (MEASURED) | 3.0 | 219 | 0.000 % |
| Gemma-4-E4B-it (MEASURED) | vLLM (assumed 6×) | 0.5 | 37 | 0.000 % | |
| Gemma-4-E4B-it (MEASURED) | SGLang (assumed 8×) | 0.4 | 27 | 0.000 % | |
| Gemma-3-12B bf16 (assumed ⅓ speed) | transformers, batch 128 (MEASURED) | 9.1 | 658 | 0.001 % | |
| Gemma-3-12B bf16 (assumed ⅓ speed) | vLLM (assumed 6×) | 1.5 | 110 | 0.000 % | |
| Gemma-3-12B bf16 (assumed ⅓ speed) | SGLang (assumed 8×) | 1.1 | 82 | 0.000 % | |
| Gemma-3-12B 4-bit quantised (assumed 0.6×) | transformers, batch 128 (MEASURED) | 5.1 | 365 | 0.001 % | |
| Gemma-3-12B 4-bit quantised (assumed 0.6×) | vLLM (assumed 6×) | 0.8 | 61 | 0.000 % | |
| Gemma-3-12B 4-bit quantised (assumed 0.6×) | SGLang (assumed 8×) | 0.6 | 46 | 0.000 % | |
| 6,000 — a unique set per voice 1,500,000 lines | Gemma-4-E4B-it (MEASURED) | transformers, batch 128 (MEASURED) | 18.3 | 1,316 | 0.002 % |
| Gemma-4-E4B-it (MEASURED) | vLLM (assumed 6×) | 3.0 | 219 | 0.000 % | |
| Gemma-4-E4B-it (MEASURED) | SGLang (assumed 8×) | 2.3 | 164 | 0.000 % | |
| Gemma-3-12B bf16 (assumed ⅓ speed) | transformers, batch 128 (MEASURED) | 54.8 | 3,947 | 0.006 % | |
| Gemma-3-12B bf16 (assumed ⅓ speed) | vLLM (assumed 6×) | 9.1 | 658 | 0.001 % | |
| Gemma-3-12B bf16 (assumed ⅓ speed) | SGLang (assumed 8×) | 6.9 | 493 | 0.001 % | |
| Gemma-3-12B 4-bit quantised (assumed 0.6×) | transformers, batch 128 (MEASURED) | 30.5 | 2,193 | 0.003 % | |
| Gemma-3-12B 4-bit quantised (assumed 0.6×) | vLLM (assumed 6×) | 5.1 | 365 | 0.001 % | |
| Gemma-3-12B 4-bit quantised (assumed 0.6×) | SGLang (assumed 8×) | 3.8 | 274 | 0.000 % |
| option | speed vs measured | note |
|---|---|---|
| Gemma-4-E4B-it | 1.0× (measured) | Already cached here and already used for the DramaBox prompt generation, so it is the known quantity. ~4 B active parameters. |
| Gemma-3-12B, bf16 | ~⅓ (assumed) | Roughly three times the active parameters. Better writing, and at these volumes the extra cost is still irrelevant. |
| Gemma-3-12B, 4-bit | ~0.6× (assumed) | Quantisation mainly buys VRAM headroom, which lets the batch grow — that is where the real gain would come from, not from the arithmetic itself. Not worth the quality risk here, since cost is not the binding constraint. |
transformers.Throughput measured 1299 tok/s at batch 128 on a single GH200; core-hours at 288 per node-hour and 4 GPUs per node; REFORMO remainder 68,709,721 core-h.