Today, a new game changer ai development is created by Chinese Ai Geeks. This is Step-Audio, moving ai talking to the next and more realistic level which is never before.
Let’s dive in…
**Step-Audio** is an open-source framework designed to unify speech comprehension and generation. It supports multilingual conversations, emotional tones, regional dialects, adjustable speech rates, and various prosodic styles, making it an advanced tool for speech-based AI applications.
Developed as an **intelligent speech interaction system**, Step-Audio boasts a **130B-parameter multimodal model** that integrates speech recognition, semantic understanding, dialogue, voice cloning, and speech synthesis. The model is accessible via Hugging Face and Modelscope repositories, making it easy for developers and researchers to use.
—
## Key Features and Innovations
### **1. 130B-Parameter Multimodal Model**
Step-Audio combines comprehension and generation capabilities in a **single, unified** model. It supports:
– **Speech Recognition**
– **Semantic Understanding**
– **Dialogue Handling**
– **Voice Cloning**
– **Speech Synthesis**
### **2. Generative Data Engine**
Step-Audio eliminates the **manual data collection** required in traditional text-to-speech (TTS) systems. Instead, it uses AI-driven generative data to enhance training quality, leading to the release of the efficient **Step-Audio-TTS-3B model**.
### **3. Granular Voice Control**
Users can fine-tune generated speech with **instruction-based controls**, adjusting:
– **Emotional tones** (e.g., anger, joy, sadness)
– **Dialects** (e.g., Cantonese, Sichuanese)
– **Vocal styles** (e.g., rap, a cappella humming)
### **4. Enhanced Intelligence**
Step-Audio uses **ToolCall mechanisms** and **role-playing capabilities** to improve performance in complex conversational scenarios.
—
## Model Overview
### **Tokenization Strategy**
Step-Audio employs a **dual-codebook framework** with **semantic** and **acoustic tokenizers**:
– **Semantic tokens:** 16.7Hz, 1024-entry codebook
– **Acoustic tokens:** 25Hz, 4096-entry codebook
– **Temporal alignment:** 2:3 ratio (2 semantic tokens per 3 acoustic tokens)
### **Language Model**
– Based on **Step-1**, a **130-billion parameter LLM**, further enhanced with **audio-contextualized pretraining** and **task-specific post-training**.
### **Speech Decoder**
The decoder converts **discrete speech tokens** into **continuous waveforms** using:
– **Flow Matching**
– **Neural Vocoding**
The **dual-code interleaving approach** ensures **smooth integration** of semantic and acoustic features.
### **Real-time Inference Pipeline**
Step-Audio optimizes real-time interactions using:
– **Voice Activity Detection (VAD)**
– **Streaming Audio Tokenizer**
– **Step-Audio Language Model & Speech Decoder**
– **Context Manager for preserving conversational continuity**
The pipeline **reduces latency** and ensures **efficient processing**.
—
## Installation and Setup
### **1. System Requirements**
– **GPU:** Minimum 1.5GB VRAM, recommended 4x A800/H800 GPUs (80GB)
– **Operating System:** Linux
– **Python:** 3.10+
– **PyTorch:** 2.3+ with CUDA
### **2. Installation Steps**
“`bash
# Clone the repository
git clone https://github.com/stepfun-ai/Step-Audio.git
cd Step-Audio
# Create a virtual environment
conda create -n stepaudio python=3.10
conda activate stepaudio
# Install dependencies
pip install -r requirements.txt
# Clone necessary model files
git lfs install
git clone https://huggingface.co/stepfun-ai/Step-Audio-Tokenizer
git clone https://huggingface.co/stepfun-ai/Step-Audio-Chat
git clone https://huggingface.co/stepfun-ai/Step-Audio-TTS-3B
“`
Once downloaded, the directory structure should be:
“`
where_you_download_dir
├── Step-Audio-Tokenizer
├── Step-Audio-Chat
├── Step-Audio-TTS-3B
“`
—
## Model Usage and Inference
### **Offline Inference**
To generate text/audio outputs from audio/text inputs:
“`bash
python offline_inference.py –model-path where_you_download_dir
“`
### **TTS Inference (Text-to-Speech)**
“`bash
python tts_inference.py –model-path where_you_download_dir –output-path where_you_save_audio_dir –synthesis-type use_tts_or_clone
“`
For **voice cloning**, a **speaker information dictionary** is required:
“`json
{
“speaker”: “speaker id”,
“prompt_text”: “content of prompt wav”,
“wav_path”: “prompt wav path”
}
“`
### **Launching a Web Demo**
For an interactive experience, start a **local web server**:
“`bash
python app.py –model-path where_you_download_dir
“`
—
## Benchmark Performance
Step-Audio demonstrates **competitive performance** across **automatic speech recognition (ASR), text-to-speech (TTS), and conversational AI** benchmarks.
### **ASR (Automatic Speech Recognition) Performance**
Compared with **Whisper Large-v3, Qwen2-Audio, MinMo, LUCY, Moshi, and GLM-4-Voice**, Step-Audio **achieves superior ASR results** in:
– **Aishell-1**
– **Aishell-2**
– **Wenetspeech**
– **Librispeech**
### **TTS Performance**
Step-Audio-TTS-3B demonstrates **best-in-class CER/WER scores**, outperforming:
– **FireRedTTS**
– **MaskGCT**
– **CosyVoice**
– **CosyVoice 2**
### **Voice Chat Capabilities**
On **StepEval-Audio-360**, a **multi-turn benchmark**, Step-Audio outperforms:
– **GLM-4-Voice**
– **Qwen2-Audio**
– **Moshi**
– **LUCY**
Scoring **higher in factuality, relevance, and chat experience**, it also excels in **role-playing, creativity, and emotional control**.
—
## Example Use Cases
### **1. Voice Cloning**
Step-Audio replicates **speaker-specific voice features**, as seen in the following examples:
| Speaker | Prompt Audio | Cloned Audio |
|———|————-|————-|
| 于谦 | [Google Drive](https://drive.google.com/file/d/1N9EJypafFwmeL0R152GoL_CVGbYn1_9A/preview) | [Google Drive](https://drive.google.com/file/d/1Zs_1QrCUuoSqtUSdn2ENIor-k5baQdDV/preview) |
| 李雪琴 | [Google Drive](https://drive.google.com/file/d/15SkZ29hksELYi1NDOxYOPu-kRTLSyke_/preview) | [Google Drive](https://drive.google.com/file/d/11Le4qMqL2DmWpf7RFRpKUXERIR9TtKC0/preview) |
### **2. Speech Speed Control**
Adjustable speech rate example:
| Prompt | Response |
|——–|———|
| **Fast Mode:** “Say a tongue twister.” | [Listen](https://drive.google.com/file/d/1mAH-NRrOVZo4tv6gdAZkyJg8kRuTNNGC/preview) |
| **Slow Mode:** “Say it again very, very slowly.” | [Listen](https://drive.google.com/file/d/1FhRnKo8uGrtO-cWg4qkrg8iDoNRbtqSX/preview) |
### **3. Emotional and Tone Control**
Step-Audio **modifies emotional tone dynamically**, e.g.:
| Prompt | Response |
|——–|———|
| “You sound robotic. Try being more expressive!” | [Listen](https://drive.google.com/file/d/19IROE6_6h2UQVNniCmDTnrhxKRMOFHq3/preview) |
### **4. Multilingual Capabilities**
Step-Audio supports **Chinese, English, and Japanese**, e.g.:
| Prompt | Response |
|——–|———|
| “What does ‘raining cats and dogs’ mean?” | [Listen](https://drive.google.com/file/d/1LEIvdR5ANMzWX8GOTqUPTNrynNS1xx–/preview) |
### **5. Rap and Singing**
Step-Audio can **generate music**, e.g.:
| Prompt | Response |
|——–|———|
| “Sing a rap song!” | [Listen](https://drive.google.com/file/d/1F8CKmVbGZ7X7d1IkQPlmndSHeG40AXha/preview) |
—
## Final Words…
Step-Audio is a **powerful and versatile speech AI framework** that sets a new standard for open-source **speech comprehension, generation, and interaction**. With robust **ASR, TTS, and voice chat capabilities**, it is an **ideal choice for developers and researchers** looking to integrate cutting-edge speech AI into their applications.
For further details, visit the [GitHub Repository](https://github.com/stepfun-ai/Step-Audio) or [Hugging Face Model Page](https://huggingface.co/stepfun-ai/Step-Audio-Chat).