StepFun AI Releases Step-Audio-R1: A New Audio LLM that Lastly Advantages from Check Time Compute Scaling

Coaching Pipeline, from Chilly Begin to Audio Grounded RL

The pipeline has a supervised chilly begin stage and a reinforcement studying stage that each combine textual content and audio duties.

Chilly begin makes use of about 5 million examples, protecting 1 billion tokens of textual content solely information and 4 billion tokens from audio paired information. Audio duties embrace computerized speech recognition, paralinguistic understanding and audio query textual content reply type dialogs. A fraction of the audio information carries audio chain of thought traces generated by an earlier mannequin. Textual content information covers multi flip dialog, information query answering, math and code reasoning. All samples share a format the place reasoning is wrapped in tags, even when the reasoning block is initially empty.

Supervised studying trains Step-Audio-R1 to comply with this format and to generate helpful reasoning for each audio and textual content. This offers a baseline chain of thought habits, however it's nonetheless biased towards textual content based mostly reasoning.

Modality Grounded Reasoning Distillation MGRD

MGRD is utilized in a number of iterations. For every spherical, the analysis group samples audio questions the place the label depends upon actual acoustic properties. For instance, questions on speaker emotion, background occasions in sound scenes or musical construction. The present mannequin produces a number of reasoning and reply candidates per query. A filter retains solely chains that meet three constraints:

1. They reference acoustic cues, not simply textual descriptions or imagined transcripts.
2. They're logically coherent as quick step-by-step explanations.
3. Their closing solutions are right in line with labels or programmatic checks.

These accepted traces kind a distilled audio chain of thought dataset. The mannequin is ok tuned on this dataset along with the unique textual content reasoning information. That is adopted by Reinforcement Studying with Verified Rewards, RLVR. For textual content questions, rewards are based mostly on reply correctness. For audio questions, the reward mixes reply correctness and reasoning format, with a typical weighting of 0.8 for accuracy and 0.2 for reasoning. Coaching makes use of PPO with about 16 responses sampled per immediate and helps sequences as much as round 10 240 tokens to permit lengthy deliberation.

Benchmarks, closing the hole to Gemini 3 Professional

On a mixed speech to textual content benchmark suite that features Huge Bench Audio, Spoken MQA, MMSU, MMAU and Wild Speech, Step-Audio-R1 reaches a mean rating of about 83.6 p.c. Gemini 2.5 Professional experiences about 81.5 p.c and Gemini 3 Professional reaches about 85.1 p.c. On Huge Bench Audio alone, Step-Audio-R1 reaches about 98.7 p.c, which is greater than each Gemini variations.

For speech to speech reasoning, the Step-Audio-R1 Realtime variant adopts pay attention whereas pondering and assume whereas talking type streaming. On Huge Bench Audio speech to speech, it reaches about 96.1 p.c reasoning accuracy with first packet latency round 0.92 seconds. This rating surpasses GPT based mostly realtime baselines and Gemini 2.5 Flash type native audio dialogs whereas protecting sub second interplay.

Ablations, what issues for audio reasoning

The ablation part supplies a number of design indicators for engineers:

• A reasoning format reward is critical. With out it, reinforcement studying tends to shorten or take away chain of thought, which lowers audio benchmark scores.
• RL information ought to goal medium problem issues. Choosing questions the place go at 8 lies in a center band provides extra secure rewards and maintains lengthy reasoning.
• Scaling RL audio information with out such choice doesn't assist. High quality of prompts and labels issues greater than uncooked dimension.

The researchers additionally describe a self cognition correction pipeline that reduces the frequency of solutions reminiscent of ‘I can solely learn textual content and can't hear audio’ in a mannequin that's educated to course of sound. This makes use of Direct Desire Optimization on curated choice pairs the place right habits is to acknowledge and use audio enter.

Key Takeaways

1. Step-Audio-R1 is without doubt one of the first audio language mannequin that turns longer chain of thought right into a constant accuracy acquire for audio duties, fixing the inverted scaling failure seen in earlier audio LLMs.
2. The mannequin explicitly targets Textual Surrogate Reasoning by utilizing Modality Grounded Reasoning Distillation, which filters and distills solely these reasoning traces that depend on acoustic cues reminiscent of pitch, timbre and rhythm as an alternative of imagined transcripts.
3. Architecturally, Step-Audio-R1 combines a Qwen2 based mostly audio encoder with an adaptor and a Qwen2.5 32B decoder that all the time generates reasoning segments earlier than solutions, and is launched as a 33B audio textual content to textual content mannequin beneath Apache 2.0.
4. Throughout complete audio understanding and reasoning benchmarks protecting speech, environmental sounds and music, Step-Audio-R1 surpasses Gemini 2.5 Professional and reaches efficiency similar to Gemini 3 Professional, whereas additionally supporting a realtime variant for low latency speech to speech interplay.
5. The coaching recipe combines giant scale supervised chain of thought, modality grounded distillation and Reinforcement Studying with Verified Rewards, offering a concrete and reproducible blueprint for constructing future audio reasoning fashions that really profit from check time compute scaling.

Editorial Notes

Step-Audio-R1 is a crucial launch as a result of it converts chain of thought from a legal responsibility into a great tool for audio reasoning by straight addressing Textual Surrogate Reasoning with Modality Grounded Reasoning Distillation and Reinforcement Studying with Verified Rewards. It exhibits that check time compute scaling can profit audio fashions when reasoning is anchored in acoustic options and delivers benchmark outcomes similar to Gemini 3 Professional whereas remaining open and virtually usable for engineers. General this analysis work turns prolonged deliberation in audio LLMs from a constant failure mode right into a controllable and reproducible design sample.

Try the Paper, Repo, Project Page and Model Weights. Be happy to take a look at our GitHub Page for Tutorials, Codes and Notebooks. Additionally, be happy to comply with us on Twitter and don’t neglect to affix our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.