Create AI Voice Agents with LiveKit
Production-grade STT→LLM→TTS pipeline. Sub-second latency. Works over SIP for inbound PSTN calls.
AI Voice Pipeline
Quick answer
What is a LiveKit voice agent?
A LiveKit voice agent is an AI agent built with the open-source LiveKit Agents framework that joins WebRTC rooms or phone calls through LiveKit SIP, listens with speech-to-text, decides with a language model and its tools, and replies with text-to-speech. CelloIP builds and self-hosts production LiveKit voice agents in any language.
Voice Agent Architecture
STT → LLM → TTS pipeline
An AgentSession with separate speech-to-text, language model and text-to-speech, all streaming. The most control over each step, voices and languages. Best for most phone agents.
Realtime speech-to-speech
A realtime model such as the OpenAI Realtime API or Gemini Live hears and speaks directly. Very low latency, with less control over voice, transcripts and each step.
Custom Pipeline
Full control over audio processing, custom VAD, tool calling, multi-modal input.
STT / LLM / TTS Providers
| Provider | Type | Latency | Quality | Self-Host | Best For |
|---|---|---|---|---|---|
| Deepgram Nova-3 | STT | Low | ★★★★★ | No | Most English phone agents |
| Whisper (faster-whisper) | STT | Medium | ★★★★ | Yes | Self-hosted, many languages |
| AssemblyAI | STT | Low | ★★★★ | No | Accuracy, diarisation |
| Azure or Google Speech | STT | Low | ★★★★ | No | Enterprise, wide language list |
| OpenAI GPT models | LLM | Low (small tiers) | ★★★★★ | No | Reliable tool calling |
| Anthropic Claude | LLM | Low (Haiku tier) | ★★★★★ | No | Careful instruction following |
| Google Gemini | LLM | Low (Flash tier) | ★★★★ | No | Speed and cost |
| Llama or Qwen | LLM | Depends on GPUs | ★★★★ | Yes | Data stays on your servers |
| ElevenLabs | TTS | Low | ★★★★★ | No | Most natural voices |
| Cartesia | TTS | Very low | ★★★★★ | No | Fastest replies |
| OpenAI TTS | TTS | Medium | ★★★★ | No | Simple set-up |
Indicative only: latency and quality vary by language, region and model version. We test candidates on your own calls before choosing.
Key Agent Capabilities
Semantic Turn Detection
Transformer model reduces interruptions and improves natural conversation flow.
Streaming Pipeline
STT→LLM→TTS overlap enables sub-second response times.
Tool/Function Calling
CRM lookup, calendar booking, DB queries, MCP tools integration.
Interruption Handling
VAD detects user barge-in, gracefully stops TTS and resets state.
Multi-language Support
Automatic language detection, seamless language switching mid-call.
DTMF + IVR Fallback
Handle keypress input when speech recognition fails.
Minimal Python Voice Agent
from livekit import agents
from livekit.agents import Agent, AgentSession
from livekit.plugins import deepgram, elevenlabs, openai, silero
class SupportAgent(Agent):
def __init__(self) -> None:
super().__init__(instructions=(
"You are a friendly customer service agent. Keep replies short. "
"If you can't help, offer to transfer the caller to a person."
))
async def entrypoint(ctx: agents.JobContext):
await ctx.connect()
session = AgentSession(
vad=silero.VAD.load(),
stt=deepgram.STT(model="nova-3"),
llm=openai.LLM(model="gpt-4o-mini"),
tts=elevenlabs.TTS(),
)
await session.start(room=ctx.room, agent=SupportAgent())
await session.generate_reply(instructions="Greet the caller.")
if __name__ == "__main__":
agents.cli.run_app(agents.WorkerOptions(entrypoint_fnc=entrypoint))SIP + Telephony Integration
Inbound Flow
PSTN Caller → SIP Carrier (Twilio/Telnyx) → LiveKit SIP Bridge → LiveKit Room → Voice Agent
Asterisk/FreeSWITCH Bridge
Caller → Asterisk PBX → SIP trunk to LiveKit → Voice Agent
Outbound Flow
Agent → LiveKit SIP → SIP Carrier → PSTN Number
Real-World Use Cases
Customer Service Bot
Handle tier-1 support 24/7, escalate complex issues to humans.
AI Sales Qualifier
Pre-screen leads, qualify intent, book qualified meetings.
Healthcare Front Desk
Appointment booking, reminders and common patient questions, with handover to staff.
Appointment Scheduler
Answer inbound calls, check calendar, confirm bookings.
A Real LiveKit Project
CelloIP built a live AI voice agent for a contact centre in North America on LiveKit Agents. The centre's FreeSWITCH platform sends calls to LiveKit over SIP; the agent handles inbound support, outbound calls and appointment booking in English and Spanish, works with the CRM, booking system and helpdesk, and hands calls back to human agents with a summary.
Read the case study →Guide ranges, matching our AI voice agent development page. Speech, model and hosting usage are separate. Step-by-step build: LiveKit voice agents guide · LiveKit with Asterisk and FreeSWITCH · self-hosted LiveKit · LiveKit vs Pipecat · LiveKit with Gemini Live
Frequently Asked Questions
What is a LiveKit voice agent?
An AI agent built with the open-source LiveKit Agents framework. It joins a WebRTC room or a phone call through LiveKit SIP, listens with streaming speech-to-text, decides what to say and do with a language model and its tools, and replies with text-to-speech, with turn detection and interruption handling built in.
Can LiveKit voice agents answer and make real phone calls?
Yes. LiveKit SIP connects to SIP trunks from carriers such as Twilio or Telnyx, or to your own Asterisk or FreeSWITCH PBX. Dispatch rules send each incoming number to the right agent, and agents can place outbound calls the same way.
How fast can a LiveKit voice agent reply?
With streaming speech-to-text, a fast language model and streaming text-to-speech hosted close together, replies typically start in about a second or less. Realtime speech-to-speech models can be faster still, with less control over each step.
Can we self-host everything?
Yes. LiveKit server, the agents and the models can all run on your own servers: for example self-hosted Whisper for speech-to-text, an open-weight language model served with vLLM, and a local text-to-speech voice. That keeps call audio and transcripts inside your infrastructure.
How long does it take and what does it cost?
As a guide, a managed-platform MVP for one use case is typically $8,000–$20,000 over 4–6 weeks, and a custom production agent on a self-hosted framework such as LiveKit $25,000–$60,000 over 8–16 weeks, depending on integrations, languages and telephony. Speech, model and hosting usage are separate running costs.
Has CelloIP built LiveKit voice agents in production?
Yes. For example, we built a live AI voice agent for a North American contact centre on LiveKit Agents, connected over SIP to the centre's FreeSWITCH platform, handling inbound support, outbound calls and appointment booking in English and Spanish. The case study describes it.
Ready to Build AI Voice Agents?
Let CelloIP's expert team integrate LiveKit voice agents into your VoIP infrastructure.