AI · AI model
Hibiki-Zero
Open-source real-time multilingual speech-to-speech translation model with voice preservation
Hibiki-Zero is a 3B-parameter open-source model developed by Kyutai for real-time speech-to-speech translation. It translates spoken French, Spanish, Portuguese, and German into English while preserving the original speaker's voice. Built using reinforcement learning to optimize the trade-off between latency and translation accuracy, it can run locally on modern consumer GPUs.
What it's for
- Translate spoken foreign languages to English in live real-time conversations
- Translate recorded multilingual audio files in batch mode
- Preserve speaker vocal characteristics and tone during cross-language translation
- Host a local speech-to-speech translation server on consumer GPU hardware
Features
- Real-time speech-to-speech translation into English
- Supports input in French, Spanish, Portuguese, and German
- Voice transfer capability that retains the original speaker's timbre and tone
- Trained via reinforcement learning for word-level alignment and low latency
- Batch inference support for pre-recorded audio files
- Built-in local real-time server with optional Gradio tunnel forwarding
- Runs on NVIDIA GPUs with 8 GB to 12 GB VRAM
Pricing
Open source. Free and open source (MIT license)
Details
- Type
- AI model
- Category
- AI
- Topic
- Speech Translation
- Open source
- Yes
- Platforms
- Linux, Command line, Self-hosted, Code library
- For
- AI researchers, audio engineers, and developers building live multilingual voice applications
- Alternative to
- SeamlessM4T, Whisper
- Tags
- speech to speech, translation, audio, voice cloning, kyutai, open source
Why it was saved
- @kyutai_labs: Real-time speech translation into English from French, Spanish, Portuguese and German, with voice transfer (keeps the speaker's own voice), low latency and high audio quality. Built on the original Hibiki architecture but trained with reinforcement learning instead of complex aligned synthetic data heuristics: only sentence-level alignment is required, word-level alignment is learned via RL, which simplifies data creation, lowers latency and scales to more languages. Reward is based on BLEU of intermediate translations with a parameter trading latency vs accuracy. Weights and reference code are open source.
Related
Open-source zero-shot text-to-speech and voice cloning model
Open-source deep learning toolkit for text-to-speech synthesis and voice cloning.
Lightweight, CPU-friendly text-to-speech model with fast voice cloning.
AI voice generator, voice cloning, and conversational agent platform
AI avatar video generation and translation platform driven by text prompts
Open full-duplex speech-to-speech conversational AI model by NVIDIA
Updated