Per-voice conditioned-speech corpus — design manual

The 40 % step is used because the published per-dimension sweeps (25/50/75/100/125 %) are monotone in the target score, so a fraction of the best dose is the documented way to ask for some rather than all of a dimension. For the 5-level and 3-level dimensions the ladder is mapped proportionally (0/1/3/4 and 0/1/1/2), which is why a hard-coded 0/2/4/6 was replaced by a fractional map.

All 57 dimensions with their measured best doses

8. Edge cases

Fourteen per language: 8 from emotion-voice-conditions/voice0.html (carrier line and λ as published) and 6 hall-of-fame genomes from moss-emotion-edgecase-evolution (a genetic search, population 24, 6 generations, elitism 8, best-of-4 per genome; mean LoRA lift +0.147). The GA genomes are used verbatim: their GENERAL text, delivery cue, inline burst tags, carrier line, λ and sampling are the winning genome, not a paraphrase.

Published cross-voice identity for these cells, worst first: fear scream 0.504 · laughter 0.525 · pain groan 0.564 · pain scream 0.592 · cold shiver 0.594 · whimpering 0.598 · crying 0.599 · tears 0.639. Loud extremes clone worst; quiet extremes clone best.

9. Sports commentator and explicitness

Sports: laion/moss-sports-commentator-lora, checkpoint real_r64_e8 at λ 1.0. The real-data-only checkpoint beat the mixed one on 11 of 12 prompts (Wilcoxon p = 0.0010; judge 1.979 vs 1.891, arousal 4.633 vs 3.914, WER 0.066 vs 0.061, n.s.), which is why real_r64_e8 is used rather than the otherwise-tempting r64_e8.

Explicitness: TTS-AGI/moss-explicitness-loras, checkpoint aesthetic_mix_r32 at λ 0.8, sampling temp 0.9 / top_p 0.95 / top_k 35 — the rank-32 aesthetic mix, which won the published raw sweep, in the 0.2–1.0 blend band. Gated by §4.

10. Exact prompt templates

The processor is driven exactly like this. instruction holds the WHOLE GENERAL: … SCRIPT: … caption; text holds ONLY the spoken words. An empty field renders as the literal string "None".

from transformers import AutoModel, AutoProcessor   # AutoModel, NOT AutoModelForCausalLM
proc = AutoProcessor.from_pretrained("laion/moss-tts-local-transformer-4.55b-voice-acting-v2",
                                     trust_remote_code=True,
                                     codec_path="OpenMOSS-Team/MOSS-Audio-Tokenizer-v2")
proc.audio_tokenizer = proc.audio_tokenizer.to("cuda").eval()
base = AutoModel.from_pretrained("laion/moss-tts-local-transformer-4.55b-voice-acting-v2", trust_remote_code=True,
                                 dtype=torch.bfloat16, attn_implementation="sdpa").cuda().eval()
# the missing audio_lm_heads.* / text_lm_head.weight warnings on load are BENIGN

msg = proc.build_user_message(text=spoken_words_only,
                              instruction=whole_GENERAL_SCRIPT_caption,
                              language="English",          # or "German"
                              reference=[ref_wav_path],
                              tokens=n_tokens)             # defaults to the WORD COUNT
batch = proc([[msg]], mode="generation")
out = net.generate(input_ids=batch["input_ids"].cuda(),
                   attention_mask=batch["attention_mask"].cuda(),
                   max_new_frames=400, do_sample=True,
                   text_temperature=0.7, text_top_k=50, text_top_p=1.0,
                   audio_temperature=T, audio_top_p=P, audio_top_k=K,
                   audio_repetition_penalty=1.1)
for m in proc.decode(out):
    if not m.audio_codes_list:          # DETECT this. Never assume a token fix worked.
        empty += 1; continue

LoRA merge scale

base_scaling = {nm: dict(m.scaling) for nm, m in pm.named_modules()
                if isinstance(m, LoraLayer)}          # captured ONCE, after loading

pm.base_model.enable_adapter_layers()
pm.base_model.set_adapter(list(keys))                  # base_model, not PeftModel
for nm, m in pm.named_modules():
    if isinstance(m, LoraLayer):
        for adapter, lam in keys.items():
            if adapter in m.scaling:
                m.scaling[adapter] = base_scaling[nm][adapter] * lam

PeftModel.set_adapter and base_model.set_adapter do not take the same argument. This pair is copied verbatim from $NB/act/code/actworker_v9.py; a hand-rewrite cost a whole nine-job run.

Three real captions from this corpus

Emotion, condition C (intense + contained), Fear:

GENERAL: A voice powerfully expressing fear, a terrified voice, trembling and breathless, shaking with fear, near screaming, in every breath. The feeling is fought down rather than shown: it is held in and only leaks through at the edges of phrases.
SCRIPT:
(speaking coolly on purpose, jaw tight, trying very hard not to sound terrified, the fear still audible underneath) "We were supposed to meet at the corner near the old station around six, but the bus was running late again."

VoiceNet, TEMP at “very high” (vn_TEMP__high@1.25):

GENERAL: A voice very high in tempo. The tempo reaches the absolute human limit of linguistic speed, resulting in a blistering, hyper-accelerated wall of sound. Phonation is extremely dense, causing individual word boundaries to completely blur into a continuous, high-velocity acoustic vibration.
SCRIPT:
(neutral, natural delivery, exactly as this voice normally speaks) "We were supposed to meet at the corner near the old station around six, but the bus was running late again."

Edge case, GA hall-of-fame genome, fear → screaming:

GENERAL: A horrified voice, blood-curdling, pushed to the extreme edge of scream.
SCRIPT:
(terrified, screaming, voice cracking with panic) "Help me. Somebody, please." <terrified scream><screaming>

All 40 evolved GENERAL lines and delivery cues

These are the manual's own evolved (BASE_P) prompts, used verbatim for the intense conditions. For the moderate conditions the intensifier is softened by a fixed substitution table (overwhelmingly→mildly, intensely→mildly, deeply→slightly, powerfully→mildly, utterly→slightly, unmistakably→faintly, viscerally→slightly, to the extreme→a little) plus one added sentence, so that A/B and C/D stay comparable in vocabulary. For the contained conditions the cue below is replaced by the family containment cue from §6 and a containment sentence is appended to GENERAL.

Two decisions about vocal bursts, stated

The burst adapters in laion/vocal-burst-lora-adapters (64 classes) are not merged in this corpus. The edge cases instead carry the GA's inline burst tags in the SCRIPT, which is what the edge-case search actually optimised. If burst adapters are added later, the measured operating points are: λ = 0.50 (presence 50.3 %, tail coverage 0.90, joint 0.448 — the peak; at 1.0 presence rises to 72 % but tail coverage collapses to 0.45 and WER to 0.57), the emotion adapter must sit at or below half the burst dose, and the tag goes inline mid-line with real words after it, never in the first or last 0.45 s. Five classes never fire at any dose: Hiss, Kissing Noises, Lip Smack, Person Whistling Playfully, Slurping Noises.

Batching by adapter

Groups are sorted by (block, emotion|dim|edge, condition|level, lang) and each worker takes a contiguous slice. That matters: with 114 dimension adapters and 40 emotion adapters, a stride slice (groups[shard::nshards]) would give a worker groups 32 apart in the sorted order, so no two consecutive groups would share an adapter and the LoRA loader would thrash on every single group. Adapters are loaded lazily with an LRU cap of 24 resident at once.

11. Ranking, per condition type

The base formula is the published one:

core  = z(genuineness) + w_blend · z(vocal-burst blend) + 1.25 · z(strength)
        [ + 1.0 · z(−VULN)   for contained conditions only ]
score = sigmoid(core / 2) · (1 − min(WER,1)) · (0.5 + 0.5 · naturalness)
        · (0.81 + 0.19 · clip(spk_sim,0,1)/0.7) · id_mult

All z-scores are taken within the group, over its 200 candidates — which is what makes the components comparable across conditions with wildly different absolute scales.

Target bands for the moderate rows, and floors for the intense rows, by family:

Naturalness = 0.6 · disfluency-band term + 0.4 · speed-band term. Disfluency: full credit 2.5–4.0, capped at 0.55 below 1.8 (robotic), decaying above 4.0 (a mess). Speed: full credit 2.4–3.3 words/s, penalised on both sides.

Identity enters as a ranking weight of 0.19 and a single hard floor at 0.4 (id_mult = 0.25 below it). Nothing is rejected for being below 0.58 or 0.68 — the 0.58 threshold only sets a resample_flag in the group statistics, because with 200 candidates the size of the pool already does the job the resample loop used to do.

12. Per-candidate metadata schema

Every candidate gets a row, including the worst-ranked one. Nothing is discarded.

13. Storage layout

out/<voice_cid>/P<profile>/
    cands-<shard>.tar                # WebDataset: <gid>.c<NNN>.mp3, one member per candidate
    cands-<shard>.parakeet.parquet   # stage-2 word timestamps, joined on audio_key
    meta-<shard>-<ts>.parquet        # one row per candidate, schema above
    groupstats-<shard>.jsonl         # one line per group: timings, empties, best-take summary
    throughput-<shard>.json          # measured gen/GPU-hour for this shard
    ref_spk_emb.npy                  # the reference ECAPA embedding, stored once
    safety_gate.json                 # the gate decision and its three inputs
    shard<NNN>.DONE                  # idempotent-resume marker

Audio is mono MP3 at 16 kHz. The tar member name is the audio_key column, so audio and metadata join without a manifest. Shards are written append-only and the worker re-reads groupstats on restart, so an interrupted job resumes at group granularity.

14. Compute estimate: 100 / 1,000 / 6,000 voices

Extrapolating at production settings (32 candidates/group), using the per-block measured rates above rather than a single average — the blocks differ by more than 3× and a single average would flatter the estimate:

One voice costs 12.16 GPU-hours at 32 candidates per group. Storage is measured at 111 kB per candidate (MP3 + parquet row including both embeddings) from the demo artefacts.

Assumptions

• Rates are measured on GH200 nodes, 4 GPUs per node, one worker process per GPU, batch 32, and include scoring (VoiceNet + EmoNet + quality + genuineness + blend + ECAPA + VoiceCLAP + whisper-large-v3-turbo). They do not include the ~2 min per-worker model load, which is amortised over a shard and adds under 2 % at these shard sizes, nor the stage-2 Parakeet pass, which runs far faster than real time and adds roughly 3–5 %.
• Rates assume the group list stays sorted so consecutive groups share adapters. With 114 dimension adapters and 40 emotion adapters, striding instead of chunking would thrash the LoRA loader on every group.
• Storage assumes 16 kHz mono MP3 and fp16 embeddings. Keeping fp32 embeddings roughly doubles the parquet size.
• The rate is measured at 200 candidates per group but applied at 32. That is defensible because the per-candidate cost is dominated by the fixed per-batch cost (batch 32, so 200 candidates is 7 batches and 32 is 1) rather than by per-group overhead: the same emotion groups measured at 32 candidates with 2 workers per node ran at ~3,300 gen/GPU-h against ~2,100 at 200 candidates with 4 workers per node. The candidate count is not what moves the number — node contention is, and the estimate deliberately uses the busier, more realistic 4-workers- per-node figure.
• No voice-conversion repair is included; it is invoked only below 0.45 similarity and costs ~1 s per repaired clip.

The weakest link

It is the voicenet block, and the reason is count, not speed. Measured per-candidate throughput is nearly flat across the blocks — sports 4,263 gen/GPU-h against emotion 2,092, a spread of only 2.04× — so cost tracks the number of groups almost exactly. The voicenet block is 56 % of the matrix and therefore 56 % of the GPU-hours.

That has a useful consequence: there is no clever optimisation hiding here. Nothing is pathologically slow, so the only real lever is generating fewer cells. Halving the voicenet block — two levels instead of four, or the 30 highest-|Δ| dimensions instead of all 57 — takes 28 % straight off the total, and no amount of batching or kernel work will do the same.

Secondary constraint: the scoring stack is 4–5 % of wall time only because the VoiceCLAP embedder is shared across the VoiceNet, genuineness and blend heads — one encode per clip instead of three. Un-sharing it would roughly triple that term.

Is 6,000 voices feasible?

At the full 808-group matrix, no — not as a single campaign. It is 72,982 GPU-hours, which at 256 GPUs is 285 hours of wall clock of uninterrupted whole-partition time, and it produces 17 TB. That is a program, not a run.

Defensible subset. Three cuts, in the order they should be taken:

1. Full matrix on 100 voices stratified over the reference pool by gender, AGEV band and DNSMOS — enough to establish that the conditioning transfers across voices. 1,216 GPU-hours.
2. Reduced matrix on 1,000 voices: 40 emotions × 2 conditions (A and C — the two that actually differ, since B and D are both λ 0.5) × 2 languages, the 20 highest-|Δ| VoiceNet dimensions × 2 extreme levels × 2 languages, all 14 edge cases, sports and explicitness = 270 groups, one third of the matrix.
3. Emotion block only on 6,000 voices if per-speaker emotion coverage is what the downstream model needs: 320 groups is 40 % of the matrix but the fastest 40 %.

15. Reproducing this from nothing

NB=/e/data1/datasets/playground/mmlaion/schuhmann1/dramabox
export HF_HOME=$NB HF_HUB_CACHE=$NB/hfcache/.cache/dramabox
# 1. LOGIN NODE ONLY - compute nodes have no internet
$NB/env_transcribe/bin/python $NB/vprof/code/prefetch.py     # datasets + LoRA repos
$NB/env_transcribe/bin/python $NB/vprof/code/vpprep.py       # voices, text pool, matrix
# 2. GPU - 16 nodes x 4 GPUs, 32 shards x 2 profiles
sbatch --array=0-15 --export=ALL,VOICE=<cid>,CANDS=32,BATCH=32,NSH=32 \
       $NB/vprof/code/vprof.sbatch
# 3. stage-2 ASR, in the NeMo venv (never pip-install nemo into env_transcribe)
$NB/env_asr/bin/python $NB/vprof/code/vpasr.py --indir $NB/vprof/out/<cid>/PA
# 4. grid + docs
$NB/env_transcribe/bin/python $NB/vprof/code/vpgrid.py --voice <cid>
$NB/env_transcribe/bin/python $NB/vprof/code/vpdocs.py --voice <cid>

Cluster constraints that are not optional. There is no system python — use $NB/env_transcribe/bin/python. Compute nodes are offline; run with HF_HUB_OFFLINE=1 after pre-fetching everything on the login node. Stagger worker startup by ~150 s on a multi-GPU node: concurrent import transformers from /e puts every worker into uninterruptible sleep. Never write to $HOME (~20 GB quota). Launch long jobs from setsid-detached scripts. Never pip install without --no-deps into this venv — it has been destroyed that way by a torch downgrade.

Bill of materials

16. What is assumed, and what could not be verified

1. The A/B/C/D stage-direction text is not published. Only the four sublabel strings exist publicly. The captions here are built from the per-emotion evolved prompt with a documented softening rule for the moderate rows and the published containment cues for the contained rows. This is a reconstruction, not the original instruction text.
2. The 4-level VoiceNet ladder is a construction. The manual has 7 ordinal levels and an adverb ladder; the mapping from “extremely low … very high” to a dose and an anchor level is defined in §7 and is ours.
3. The 40 % moderate dose rests on the published sweeps being monotone. It was not separately validated at 40 %.
4. The finetranslations domain taxonomy is derived, not given. subsets.csv is a language table. The URL-host + keyword labeller is ours and is stored per row so it can be replaced.
5. German carriers for the edge cases are fixed translations written into the spec, not drawn from finetranslations, because for those groups the carrier line is the content.
6. The emotion family map is ours. The published material names five families and gives recipes for a handful of anchor emotions; assigning all 40 emotions to a family is an extrapolation, and Helplessness in particular sits between the fear and sadness families.
7. The cognitive family's target band [0.5, 1.0] is marked “extrapolated” in the source and is carried forward as such.
8. In-loop WER uses whisper-large-v3-turbo, not Parakeet v3. Parakeet lives in a different venv (env_asr, NeMo 2.7.3) and installing NeMo into the generation venv risks a torch downgrade that has destroyed it before. Parakeet runs as stage 2 and supplies the word timestamps; because every ranking component is stored, re-ranking on Parakeet WER needs no regeneration.
9. The Mediathek checkpoint used is laion/moss-mediathek-emotion-lora :: r64_e2 at 0.25, as specified. Note a second, differently named repo laion/moss-mediathek-hq-lora (r64_epoch3) also exists and is the one the voice-acting manual names; they were not compared.
10. No listener study was run. Every number here is from the local sensor stack. The published post-mortems are explicit that the composite tracks a listener at only ρ ≈ +0.21 on spoken material and negatively on non-verbal material, so the ranking should be treated as a strong filter, not as ground truth.