SteerTTS: A 9.6M-Parameter Voice You Steer With Numbers

Voice agents have a quiet split brain. The language model decides what to say, then hands plain text to a TTS that decides, on its own, how to say it. Even when the agent can tell the caller just got bad news, it has no way to ask for a slower, softer reply. When it is asking a question, it cannot ask for the rise at the end.

The options for a real-time agent are thin. Small open TTS models are fast, but take little direction beyond a voice and a speed. Instruction-following TTS takes direction in plain English, but at hundreds of millions to billions of parameters every turn costs GPU time or a metered API call, and nothing tells you whether “a bit warmer” actually came out warmer.

We wanted a small, fast TTS that an agent can steer turn by turn, using knobs we can check. We could not find one, so we built one. SteerTTS is a 9.6M-parameter English acoustic model (23M with its vocoder). Every request is a set of up to 12 acoustic controls, each measured in units of the chosen voice’s own neutral speech. The same code that labels the training data reads those numbers back from the output, so whether a request worked is something you can measure.

Below: seven examples you can listen to, an interactive explorer, a support call steered turn by turn, the architecture and training, held-out results, and what does not work yet. Weights and code: huggingface.co/WhissleAI/steer-tts (weights CC BY-NC 4.0, code MIT). Training recipe: PromptingNemo/recipes/steer_tts.

Listen first

The first reply in our launch thread was blunt: “very fast, but computerized voice.” Fair. A 9.6M-parameter model will not match a billion-parameter one on naturalness, and the voice you pick matters a lot. v1.0.1 moved the default from voice_1361 to voice_1688: same WER, higher UTMOS22 (4.43 vs 4.24), a little slower. We picked it from 300 random voices on our own sentences, not on any test set. The other 2,823 are in voices.json, with measured descriptions.

Now the point of the model: the same voice, steered.

The idea: steering is a measurement problem first

Instruction-following TTS has mostly scaled up the language side. Models read a prompt such as “a warm, slightly hurried voice” through a large text encoder, or put the TTS inside an LLM backbone. They are impressive, but they are large, and it is hard to say whether a given output did what was asked. Each paper uses its own classifier and its own test set, which is why so few controllable-TTS numbers compare across papers.

We went the other way. If a tiny network has to understand open-ended language and render speech, it does both badly. So we stopped asking it to understand language. We defined a small set of speaker-relative acoustic quantities that we can measure on any clip, and made the model a faithful renderer of those numbers. Deciding which delivery a reply needs happens outside the model: in a 0.5M text adapter, in rules you write, or in the agent’s LLM.

This choice makes the whole system checkable. A request of {"loud": 2} means “two within-speaker standard deviations louder than this voice’s neutral speech.” After rendering, we run the same measurement on the output and see how far it moved. No learned judge sits between the request and the verdict.

Where the control space came from

Before building anything, we ran a parallel-delta study on CREMA-D (91 actors, 12 sentences, 7,442 clips). Every emotional take was compared with the same actor’s neutral take of the same sentence, so speaker and text cancel out. Three findings shaped the design:

• Humans are the bottleneck, not acoustics. Voice-only crowd raters agreed with the acted label on only 41.5% of clips. 67% of sad clips were heard as neutral. Emotion labels are a weak target. Measurable acoustics are a strong one.
• Arousal is acoustic; valence mostly is not. Loudness, tilt and pitch separate emotions by arousal, but happy and angry at equal arousal barely differ acoustically. Valence has to come from an explicit category (our style tags) or from the words, never from a continuous acoustic knob.
• Speakers differ fourfold in expressiveness. A raw “+6 dB” means different things for different people. Hence speaker-relative units.

The 12 controls

Each control is an utterance-level measurement, computed on speech frames between the first and last speech frame. It is z-scored as z = (x − μ_{speaker, neutral}) / σ_pooled: the speaker’s own neutral mean, and the pooled within-speaker SD across all voices, clipped to ±5. Because σ is pooled, “+1 loud” is the same physical step for every voice, about 1.9 dB, taken from that voice’s own baseline.

The mask matters as much as the values. The model receives the 12 values and a 12-bit mask of which ones were specified. {"rate": 0} asks for this voice’s neutral rate. Leaving rate out lets the model choose it from the voice, the tag and the text, so a fast tag can still speed things up. During training each control is masked with probability 0.35, all of them with probability 0.15, and the tag is dropped with probability 0.3. Partial requests are therefore in-distribution, and the same dropout trains the unconditional branch that guidance needs.

Worked examples

Every clip below is a fresh render of the released model (flow refiner on, seed 0, laptop CPU). Each “measured” value is the readback: the labelling code run on that clip, compared with the same voice reading the same text with no request. Single clips are noisy; Figure 3 has the statistics.

Example 1. What “+1” means, in decibels and semitones

Because the SD is pooled across speakers, “+1” is the same relative step for every voice. For pitch it is +2.09 semitones: about a whole tone, which is a different number of hertz for a low voice and a high one. For loudness it is +1.93 dB. Nobody has to know a voice’s baseline to ask for “a bit higher”; the model starts from wherever that voice already is.

Example 2. One sentence, six knobs

Pick a control and a level, and press play. The neutral render is in the middle. Each button is a separate render at guidance 1.5 (the setting that roughly delivers the requested size), and the readout shows what the measurement code read back from that clip. Not every clip lands: f0_iqr at −2 moves the wrong way here, which is what a .66 held-out hit rate (Table 7) looks like on one clip.

Example 3. Statement or question? One number.

In English, a statement becomes a question through the pitch of its last few hundred milliseconds. f0_end is exactly that measurement: the slope of F0 over the final 300 ms of voicing. An agent that is asking for confirmation can say so with one number.

Example 4. Same request, three voices

Do controls transfer across voices, or do you retune per voice? Here is one “slower, lower, softer” request, {"rate": -1.5, "f0_med": -1, "loud": -1}, sent unchanged to three very different voices.

Example 5. “Unspecified” is not “zero”

Leaving a control out and setting it to 0 are different requests. Here a fast tag meets the rate control:

Example 6. Guidance, by ear

Guidance (explained under Training) scales up a request at inference time. It buys size and costs cross-talk:

Example 7. One sentence, four meanings

The classic linguistics example: move the stress in “I didn’t say she took the money” and the meaning moves with it. Markup *word* asks for +2.5 semitones, +4 dB and 20% more duration on one word, inside a single forward pass, so the rest of the sentence stays continuous around it.

Architecture

SteerTTS is a non-autoregressive, explicit-variance model in the FastSpeech 2 / FastPitch lineage. Text becomes ARPAbet phones with word indices. An encoder contextualises them. Three small predictors set each phone’s duration, pitch and energy. A length regulator expands phones to frames, and a decoder produces a 100-band log-mel that the pretrained Vocos vocoder turns into 24 kHz audio. We chose explicit variance on purpose: when duration, pitch and energy are separate, inspectable predictions, steering has somewhere concrete to act, and per-word edits are just additive shifts on the right tokens.

Three choices are less standard:

• Conditioning everywhere, through one vector. Voice, tag and the masked controls are combined by a small MLP into a 128-d vector g. Every LayerNorm in the encoder, decoder and predictors is adaptive: its scale and shift are predicted from g, with zero init, in the style of DiT. Nothing is concatenated to the input and hoped to survive eight blocks.
• Hybrid blocks: attention plus bidirectional Mamba-3. Each of the 4 + 4 blocks runs 2-head self-attention, then a bidirectional Mamba-3 branch (forward and backward scans summed; complex-valued state, trapezoidal discretisation), then a conv FFN. We did not train the hybrid from scratch. We grafted zero-initialised SSM branches into a trained attention-only model, so the network was exactly the same function at step zero, then fine-tuned it. The Mamba branches are 3.70M of the 9.59M parameters.
• Voice from a speaker encoder, not a lookup table. The voice slot is a linear projection of a frozen 512-d WavLM-base-plus-sv x-vector. During training each clip is conditioned on the x-vector of a different, neutral utterance by the same speaker. The vector therefore has to carry identity, not the target clip’s style, which keeps voice and steering disentangled.

Two optional decoders sit on top. The L1-trained decoder predicts something like a conditional median, which comes out slightly hazy in the upper bands. The flow refiner (17.6M, OT-CFM on a ConvNeXt with AdaLN-zero, 10 midpoint steps) samples that missing detail given the base mel and g. Every control still comes from the acoustic model. The prompted flow (24.7M) adds attention and is given the reference clip’s mel as context, Voicebox-style; it is used only for cloning. We remove the base mel’s static spectral envelope before the flow sees it, so the voice colouring must come from the prompt.

Training

The final training set has 410,566 utterances (659 h) from 4,919 speakers. Most of it is clean read speech, which teaches the model to talk. The expressive corpora teach it to move.

The base loss is standard: L1 on the mel, MSE on log-duration and token pitch/energy, and a RAD-TTS-style aligner (forward-sum CTC with monotonic alignment search) trained jointly. The model grew in stages, each initialised from the last. It started as an attention-only model on 25 EARS speakers. Then came more corpora, the Mamba graft, x-vector conditioning, and the full data, and finally two stages of fine-tuning aimed squarely at steering. The released checkpoint is step 312.5k. Four details made the difference:

1. Level normalisation

Speaker x-vectors are, by design, invariant to recording level. EARS is recorded about 19 dB quieter than LibriTTS-R. Without a fix, the model had no way to infer a voice’s level from its vector. It rendered every voice at the loud corpora’s level, and the quiet ones clipped. The fix is to shift each target mel so that every speaker’s neutral speech sits at the same level. Loudness then lives only where it belongs: in loud.

2. A paired-intervention loss

Supervised training on noisy labels teaches correlation, not causation. A clip labelled “fast” is also different in a dozen other ways. Our corpora contain natural interventions: the same speaker reading the same text under different conditions. Examples are EARS reading styles, Expresso styles, and synthetic speed, volume and emotion variants. For each such pair we ask the model’s predicted change between the two conditions to match the real change, per token (log-duration, pitch, energy) and in the mean spectrum. This trains the direction of a control directly, with speaker and text held fixed.

3. Reward fine-tuning, with a leak penalty and an anchor

Every fourth step, the model renders a few steered requests end to end, through the frozen vocoder’s differentiable ISTFT. A small readback network (1.0M, trained on real mels) measures what came out. The reward combines several terms. Control asks whether the requested dims moved. Leak asks whether the un-requested dims stayed where the unsteered render put them, which stops the model from gaming the reward by moving everything. Intelligibility is the CTC loss under a frozen wav2vec2. Quality is UTMOS. Identity is speaker similarity. A trust-region anchor keeps unsteered renders close to the pre-fine-tuning model. Reward and supervised gradients are combined with PCGrad.

In honesty: the two objectives fight. About 70% of reward steps conflicted with the supervised gradient. An emotion-probe reward we added never got satisfied (it stayed near 3 nats, below chance). The control, leak and anchor terms are the ones that helped.

4. Guidance, because noisy labels shrink effects

Control labels are measurements, and measurements are noisy. A regression trained on noisy inputs learns an attenuated slope. Statisticians call this regression dilution. We see exactly that: ask for +2 and you get about +1 (mean gain ≈ 0.5). We correct it at inference with classifier-free-style guidance. The model renders the request and the same voice with no tag and no controls, then extrapolates: out = out₀ + s · (out − out₀). It does this on durations, pitch, energy and the mel, so both passes share one timeline.

Four ways to ask

Every synthesis is one JSON object, documented in a JSON Schema that ships with the model. Only text is required. On top of the numbers there are three friendlier layers. Each one resolves to the same request, so you can always see what the model was actually asked to do.

from steertts import SteerTTS

tts = SteerTTS.from_pretrained("WhissleAI/steer-tts")  # GPU if available

# 1. Numbers: z-scores against this voice's neutral speech
wav = tts("Are you serious?", ctrl={"f0_end": 2.0, "f0_iqr": 1.5})

# 2. A style tag (51 of them), optionally with guidance
wav = tts("It is over.", tag="sadness", ctrl={"loud": -1}, guidance=1.5)

# 3. Free text -> an inspectable request
req = tts.resolve({"text": "See you soon.", "prompt": "excited, a bit faster"})
print(req)  # {'tag': 'excited', 'ctrl': {'rate': 0.29}, ...}
wav = tts.synthesize(req)

# 4. Inline markup, streamed sentence by sentence
text = "Well, *that* was fast. [pause 400ms] <slow>Let me think.</slow>"
for chunk in tts.stream(text):
    play(chunk.audio)  # plays while the rest renders

• Free-text directions go through a 0.5M adapter on a frozen MiniLM sentence encoder. It predicts a tag, which controls are specified, and their values. It was trained on captions generated from measured values: a phrase such as “much slower” is emitted only when that clip’s measured |z| is large, and its grade follows |z|. On 161 hand-written prompts it picks the right dims with precision .85 and recall .66, gets the sign right 88% of the time, and gets the tag top-1 56% of the time. It fails on composition and negation (“not fast” can come out fast). Always look at the resolved request.
• Markup is *emphasis*, <slow> <fast> <high> <low> <loud> <soft>, exact pauses such as [pause 300ms], and style spans such as <whisper>…</whisper>. Per-word edits are applied inside a single forward pass, so prosody stays continuous; neighbouring words move by at most 0.19 st. Markup pauses are exact, so prefer them to the weak pause control.
• Streaming splits at sentence ends, pauses and style boundaries and yields audio in order. First audio arrives about 0.5 s after the call on a loaded 4-thread CPU, and 60–85 ms on a warm GPU.

How well it steers

Everything below is held-out: speakers that share no speaker, chapter or recording with the training data. Readback uses the same code that made the labels, plus Praat as an independent second extractor. These numbers are for the acoustic model at guidance 1, without the refiner.

Six controls are reliable: loudness, median pitch, H1–H2, tilt, brightness and pitch range all track their requests with ρ ≥ .85 under both extractors. Rate and final rise/fall are usable; their ranking depends on which extractor you trust. Pause, voiced share and CPPS are weak. CPPS is not a working control at all: requesting it moves pitch and level about as much as CPPS itself. Mean leakage into the other 11 dims is 0.15–0.30 SD. Steering costs nothing in intelligibility: WER (Whisper-large-v3) is 0.94% uninstructed and 0.84–1.01% across all requests.

Does a +2 mean the same thing on a voice it has never heard?

This is the property that makes speaker-relative units worth their trouble. To test it, we re-ran the protocol at a larger scale. We used 62 unseen voices: 31 from LibriTTS-R test-clean and 31 from GLOBE, a corpus the model never trained on. Each voice got 16 sentences, 12 of them newly written, for 992 renders per direction per control. Confidence intervals come from a crossed voice × sentence bootstrap. One caveat: z = 0 means the voice’s typical render, which is close to, but not the same as, its real recordings.

Intelligibility, quality and speed

Intelligibility is strong for the size. Cloning similarity is not. A SIM of 33 sits below the board’s lowest entry (about 37), and the median is about 65. Don’t expect a convincing likeness. The vocoder is not the bottleneck: Vocos copy-synthesis of the real targets scores 68.4 against 72.0 for the real audio. On ParaSpeechCaps test prompts, WER is 3.07 under the same ASR the ParaSpeechCaps authors used, against 4.62 reported for Parler-TTS. Our objective accuracy on those prompts’ pitch-level and rate attributes is lower: about 0.6, against about 0.9 for real speech.

What doesn’t work yet

We would rather you hear this from us than find it in production.

There are also experiments we tried and dropped. A GAN fine-tune of the vocoder on flow mels left the discriminators at chance. A speaker-similarity fine-tune of the prompted flow did not help. The weights contain an acoustic-environment input (scene, room, phone), but the final stage never trained it, so the package does not expose it.

How we tried to keep ourselves honest

• Circular judges are labelled as such. UTMOS, the WavLM-base speaker encoder and our emotion probe were training rewards, so we do not use them for claims. Our claims rest on the DSP readback, Praat, Whisper and Qwen3 ASR, WavLM-large SIM and emotion2vec, none of which was optimised against. (For example, SIM measured with the conditioning encoder reads 0.925. Measured with an independent one, it reads 0.27–0.35.)
• The DSP readback measures consistency with the label definition, not perception. A control can hit its number and still not sound the way you would describe it in words.
• Some choices saw the test sets. Text-normalisation rules came from CV3-Eval error analysis, and the prompted-flow checkpoint was chosen on subsets of Seed-TTS and CV3. The leaderboard voice is the pre-registered one, not the lowest-WER voice from our sweep.
• No human evaluation, and one training seed. Every number in this post is automatic. A proper listening study is the next thing to do.
• Bit-exact release. The released package reproduces the training code’s output bit for bit on the same device across seven cases, from acoustic-only through cloning. The verification log ships with the weights.

Driving it from an agent

A request is a small, schema-validated JSON object, so it fits in a tool call. Whatever decides the delivery (the agent’s LLM, a rules layer, or per-intent presets a designer writes) emits a few bounded numbers that validate against request.schema.json and are easy to log. Which of those makes the best choices is an open question. We have not found an LLM that reliably picks good deliveries on its own, and we have not benchmarked it. SteerTTS is the other half: once something decides, the voice carries it out, and you can check that it did.

Here is what that looks like on a real conversation. A customer calls about a lost package. We wrote one delivery per agent turn by hand, the way a rules layer or a well-prompted LLM might, and rendered the four turns twice: flat, the way most agents speak today, and steered.

Most of it lands. Turn 2 slows and softens on cue, and turn 3 rises like a question. Turn 4 mostly misses: the emphasis on tomorrow stretches one word, which drags the measured rate down, and a single clip’s pitch range is the noisiest of the reliable controls. The readback catches exactly this, turn by turn, before a customer does.

The glue code is small:

# Your agent returns, per sentence (LLM, rules, or both):
# {"text": "I'm sorry, that order didn't go through.",
#  "ctrl": {"rate": -0.8, "loud": -0.5, "f0_end": -1.0},
#  "tag": "none"}
turn = agent_reply(conversation)
for sentence in turn["sentences"]:
    req = {"text": sentence["text"], "voice": "voice_1688",
           "ctrl": sentence.get("ctrl", {}), "tag": sentence.get("tag", "none")}
    for chunk in tts.stream(req["text"], voice=req["voice"],
                            ctrl=req["ctrl"], tag=req["tag"]):
        play(chunk.audio)

This is a pattern, not a product. The renderer is the part we built. Choosing the right delivery for a turn is still open, which is why an explicit request helps: you can log it, review it, and improve whatever wrote it.

Licence and responsible use

Weights are CC BY-NC 4.0 (non-commercial), because part of the training data is non-commercial. The steertts package is MIT. Intended uses are research on controllable and interpretable synthesis, accessibility and prototyping where explicit control matters more than peak naturalness, and on-device experiments.

Clone only voices whose owners have agreed. Do not use the model to impersonate real people, produce deceptive or fraudulent audio, bypass voice authentication, or create non-consensual content. Disclose synthetic speech to listeners; there is no watermark yet. Built-in voices are training speakers under anonymised names. Speakers from a celebrity-interview corpus were excluded from the voice list.

Open questions

• Do listeners agree with the readback? A paired “which is more X?” study per control would tell us which measured moves are also perceived moves.
• Can a tiny model render emotion, not just arousal? Our emotion reward never converged. It may be a capacity limit, a weak probe, or the valence finding from CREMA-D showing up again.
• Hybrid or pure attention? We grafted Mamba-3 into a trained attention model rather than running a controlled comparison. A clean ablation at this scale is still missing.
• Who should choose the delivery? Benchmarking how well LLMs pick deliveries, and training a small policy against the readback, is the natural next step.
• Word-level steering while streaming, which needs a causal acoustic model.

If you try it, we want to hear where the steering feels off and which controls you actually reach for. Open an issue at github.com/WhissleAI/steer-tts or start a discussion on the Hugging Face page.

References

1. A. Łańcucki. FastPitch: Parallel Text-to-speech with Pitch Prediction. arXiv:2006.06873
2. Y. Ren et al. FastSpeech 2: Fast and High-Quality End-to-End Text to Speech. arXiv:2006.04558
3. R. Badlani et al. One TTS Alignment To Rule Them All. arXiv:2108.10447
4. A. Gu, T. Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv:2312.00752. The Mamba-3 layer used here follows the Mamba-3 formulation (trapezoidal discretisation, complex state).
5. W. Peebles, S. Xie. Scalable Diffusion Models with Transformers (adaptive LayerNorm-zero). arXiv:2212.09748
6. S. Chen et al. WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing. arXiv:2110.13900
7. H. Siuzdak. Vocos: Closing the Gap Between Time-Domain and Fourier-Based Neural Vocoders. arXiv:2306.00814
8. Y. Lipman et al. Flow Matching for Generative Modeling. arXiv:2210.02747; A. Tong et al. Improving and Generalizing Flow-Based Generative Models with Minibatch Optimal Transport. arXiv:2302.00482
9. M. Le et al. Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale. arXiv:2306.15687
10. J. Ho, T. Salimans. Classifier-Free Diffusion Guidance. arXiv:2207.12598
11. T. Yu et al. Gradient Surgery for Multi-Task Learning (PCGrad). arXiv:2001.06782
12. D. Lyth, S. King. Natural Language Guidance of High-Fidelity Text-to-Speech with Synthetic Annotations (Parler-TTS). arXiv:2402.01912
13. A. Diwan et al. Scaling Rich Style-Prompted Text-to-Speech Datasets (ParaSpeechCaps). arXiv:2503.04713
14. H. Cao et al. CREMA-D: Crowd-sourced Emotional Multimodal Actors Dataset. IEEE Trans. Affective Computing, 2014.
15. Y. Koizumi et al. LibriTTS-R. arXiv:2305.18802; J. Richter et al. EARS. arXiv:2406.06185; T. A. Nguyen et al. Expresso. arXiv:2308.05725
16. Open TTS Leaderboard. huggingface.co/spaces/hf-audio/open_tts_leaderboard