Development · Tip
Open-Source Real-Time Voice Agent Stack
A low-cost, open-source stack for building real-time AI voice assistants
This recipe pairs LiveKit's VoicePipelineAgent with Groq-hosted Whisper and Llama 3.3-70B, using self-hosted Kokoro TTS for speech generation. It provides a drop-in, significantly cheaper alternative to commercial voice APIs like ElevenLabs while maintaining low latency.
How it works
Stack: TTS/voice = Kokoro (open source, OpenAI TTS-compatible endpoint), Chat/LLM = Groq Llama 3.3-70B Versatile, STT = Whisper large-v3 via Groq, orchestration = LiveKit VoicePipelineAgent; coded with Aider + DeepSeek.
Code (from photo): kokoro_tts = openai.TTS(model="tts-1" (kept for LiveKit compatibility), voice="am_michael" (voices listed at http://localhost:8880/v1/audio/voices), api_key="not-needed", base_url="http://localhost:8880/v1") pointing at a Kokoro FastAPI endpoint. agent = VoicePipelineAgent(vad=ctx.proc.userdata["vad"], stt=openai.STT.with_groq(model="whisper-large-v3"), llm=openai.LLM.with_groq(model="llama-3.3-70b-versatile"), tts=kokoro_tts, chat_ctx=initial_ctx); agent.start(ctx.room, participant); await agent.say("Hey, how can I help you today?", allow_interruptions=True). cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint, prewarm_fnc=prewarm)).
What it's for
- Build low-latency conversational voice bots without expensive proprietary TTS APIs
- Self-host text-to-speech using an OpenAI-compatible Kokoro endpoint
- Orchestrate real-time speech-to-speech pipelines using LiveKit and Groq
Key points
- LiveKit VoicePipelineAgent orchestration for handling WebRTC audio streaming and turn-taking
- Self-hosted Kokoro TTS exposed via an OpenAI-compatible FastAPI endpoint
- Whisper large-v3 hosted on Groq for ultra-fast speech-to-text
- Llama 3.3-70B Versatile on Groq for conversational reasoning
- Native support for interruption handling via LiveKit voice activity detection (VAD)
Pricing
Open source. Free and open-source stack; pay only for Groq API usage and local or cloud compute.
Details
- Type
- Tip
- Category
- Development
- Topic
- Voice AI
- Open source
- Yes
- Platforms
- Self-hosted, Linux, macOS, Windows
- For
- Developers building real-time voice agents looking to reduce API costs.
- Alternative to
- ElevenLabs Conversational AI, Vapi, Retell AI
- Tags
- voice ai, livekit, kokoro, groq, tts, stt, llm
Why it was saved
- @TheArk_Master: Concrete, copyable stack and code for building a voice agent without ElevenLabs.
Compare
Related
Platform for building and deploying interactive voice AI agents.
Open-source Python framework for building realtime voice and multimodal AI assistants
Open-source emotive text-to-speech foundation model with low conversational latency.
Platform and SDK for building custom AI voice agents and smart devices
No-code platform to build, test, and deploy conversational voice AI agents
Turn websites and documents into clean, LLM-ready markdown or structured data.
Updated