StepFun AI Releases Step-Audio-EditX: A New Open-Supply 3B LLM-Grade Audio Modifying Mannequin Excelling at Expressive and Iterative Audio Modifying

How can speech modifying turn into as direct and controllable as merely rewriting a line of textual content? StepFun AI has open sourced Step-Audio-EditX, a 3B parameter LLM based mostly audio mannequin that turns expressive speech modifying right into a token stage textual content like operation, as a substitute of a waveform stage sign processing activity.

Why builders care about controllable TTS?

Most zero shot TTS programs copy emotion, type, accent, and timbre straight from a brief reference audio. They’ll sound pure, however management is weak. Model prompts in textual content assist just for in area voices, and the cloned voice usually ignores the requested emotion or talking type.

Previous work tries to disentangle components with additional encoders, adversarial losses, or complicated architectures. Step-Audio-EditX retains a comparatively entangled illustration and as a substitute adjustments the info and publish coaching goal. The mannequin learns management by seeing many pairs and triplets the place textual content is fastened, however one attribute adjustments with a big margin.

Structure, twin codebook tokenizer plus compact audio LLM

Step-Audio-EditX reuses the Step-Audio twin codebook tokenizer. Speech is mapped into two token streams, a linguistic stream at 16.7 Hz with a 1024 entry codebook, and a semantic stream at 25 Hz with a 4096 entry codebook. Tokens are interleaved with a 2 to three ratio. The tokenizer retains prosody and emotion info, so it isn’t totally disentangled.

On high of this tokenizer, the StepFun analysis group builds a 3B parameter audio LLM. The mannequin is initialized from a textual content LLM, then skilled on a blended corpus with a 1 to 1 ratio of pure textual content and twin codebook audio tokens in chat type prompts. The audio LLM reads textual content tokens, audio tokens, or each, and at all times generates twin codebook audio tokens as output.

A separate audio decoder handles reconstruction. A diffusion transformer based mostly circulate matching module predicts Mel spectrograms from audio tokens, reference audio, and a speaker embedding, and a BigVGANv2 vocoder converts Mel spectrograms to waveform. The circulate matching module is skilled on about 200000 hours of top of the range speech, which improves pronunciation and timbre similarity.

Giant margin artificial knowledge as a substitute of difficult encoders

The important thing thought is giant margin studying. The mannequin is publish skilled on triplets and quadruplets that hold textual content fastened and alter just one attribute with a transparent hole.

For zero shot TTS, Step-Audio-EditX makes use of a top quality in home dataset, primarily Chinese language and English, with a small quantity of Cantonese and Sichuanese, and about 60000 audio system. The information covers extensive intra speaker and inter speaker variation in type and emotion.(arXiv)

For emotion and talking type modifying, the group builds artificial giant margin triplets (textual content, audio impartial, audio emotion or type). Voice actors file about 10 second clips for every emotion and magnificence. StepTTS zero shot cloning then produces impartial and emotional variations for a similar textual content and speaker. A margin scoring mannequin, skilled on a small human labeled set, scores pairs on a 1 to 10 scale, and solely samples with rating not less than 6 are saved.

Paralinguistic modifying, which covers respiratory, laughter, crammed pauses and different tags, makes use of a semi artificial technique on high of the NVSpeech dataset. The analysis group builds quadruplets the place the goal is the unique NVSpeech audio and transcript, and the enter is a cloned model with tags faraway from the textual content. This provides time area modifying supervision with no margin mannequin.

Reinforcement studying knowledge makes use of two desire sources. Human annotators price 20 candidates per immediate on a 5 level scale for correctness, prosody, and naturalness, and pairs with margin larger than 3 are saved. A comprehension mannequin scores emotion and talking type on a 1 to 10 scale, and pairs with margin larger than 8 are saved.

Submit coaching, SFT plus PPO on token sequences

Submit coaching has two phases, supervised tremendous tuning adopted by PPO.

In supervised tremendous tuning, system prompts outline zero shot TTS and modifying duties in a unified chat format. For TTS, the immediate waveform is encoded to twin codebook tokens, transformed to string type, and inserted into the system immediate as speaker info. The person message is the goal textual content, and the mannequin returns new audio tokens. For modifying, the person message contains unique audio tokens plus a pure language instruction, and the mannequin outputs edited tokens.

Reinforcement studying then refines instruction following. A 3B reward mannequin is initialized from the SFT checkpoint and skilled with Bradley Terry loss on giant margin desire pairs. The reward is computed straight on twin codebook token sequences, with out decoding to waveform. PPO coaching makes use of this reward mannequin, a clip threshold, and a KL penalty to stability high quality and deviation from the SFT coverage.

Step-Audio-Edit-Take a look at, iterative modifying and generalization

To quantify management, the analysis group launched Step-Audio-Edit-Take a look at. It makes use of Gemini 2.5 Professional as an LLM as a choose to judge emotion, talking type, and paralinguistic accuracy. The benchmark has 8 audio system, drawn from Wenet Speech4TTS, GLOBE V2, and Libri Mild, with 4 audio system per language.

The emotion set has 5 classes with 50 Chinese language and 50 English prompts per class. The talking type set has 7 types with 50 prompts per language per type. The paralinguistic set has 10 labels similar to respiratory, laughter, shock oh, and uhm, with 50 prompts per label and language.

Modifying is evaluated iteratively. Iteration 0 is the preliminary zero shot clone. Then the mannequin applies 3 rounds of modifying with textual content directions. In Chinese language, emotion accuracy rises from 57.0 at iteration 0 to 77.7 at iteration 3. Talking type accuracy rises from 41.6 to 69.2. English reveals related habits, and a immediate fastened ablation, the place the identical immediate audio is used for all iterations, nonetheless improves accuracy, which helps the big margin studying speculation.

The identical modifying mannequin is utilized to 4 closed supply TTS programs, GPT 4o mini TTS, ElevenLabs v2, Doubao Seed TTS 2.0, and MiniMax speech 2.6 hd. For all of them, one modifying iteration with Step-Audio-EditX improves each emotion and magnificence accuracy, and additional iterations proceed to assist.

Paralinguistic modifying is scored on a 1 to three scale. The typical rating rises from 1.91 at iteration 0 to 2.89 after a single edit, in each Chinese language and English, which is similar to native paralinguistic synthesis in robust business programs.

Key Takeaways

1. Step Audio EditX makes use of a twin codebook tokenizer and a 3B parameter audio LLM so it could possibly deal with speech as discrete tokens and edit audio in a textual content like means.
2. The mannequin depends on giant margin artificial knowledge for emotion, talking type, paralinguistic cues, pace, and noise, reasonably than including additional disentangling encoders.
3. Supervised tremendous tuning plus PPO with a token stage reward mannequin aligns the audio LLM to comply with pure language modifying directions for each TTS and modifying duties.
4. The Step Audio Edit Take a look at benchmark with Gemini 2.5 Professional as a choose reveals clear accuracy features over 3 modifying iterations for emotion, type, and paralinguistic management in each Chinese language and English.
5. Step Audio EditX can publish course of and enhance speech from closed supply TTS programs, and the complete stack, together with code and checkpoints, is accessible as open supply for builders.

Step Audio EditX is a exact step ahead in controllable speech synthesis, as a result of it retains the Step Audio tokenizer, provides a compact 3B audio LLM, and optimizes management by giant margin knowledge and PPO. The introduction of Step Audio Edit Take a look at with Gemini 2.5 Professional as a choose makes the analysis story concrete for emotion, talking type, and paralinguistic management, and the open launch lowers the barrier for sensible audio modifying analysis. Total, this launch makes audio modifying really feel a lot nearer to textual content modifying.

Take a look at the Paper, Repo and Model Weights. Be happy to take a look at our GitHub Page for Tutorials, Codes and Notebooks. Additionally, be happy to comply with us on Twitter and don’t neglect to affix our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.