Building AI Voice Agents with LiveKit
Complete 2026 Guide

1. Voice Agent Pipeline Architecture

A LiveKit voice agent's core is the VoicePipelineAgent — a high-level abstraction that orchestrates three streaming stages:

The key innovation is streaming overlap: STT sends partial transcripts to the LLM before the user finishes speaking; the LLM streams tokens to TTS before generating a complete response; TTS begins synthesising audio before receiving all tokens. This parallel execution cuts latency from 2–4 seconds (sequential) to under 600ms in production.

2. Installation and First Voice Agent

Install the LiveKit Agents SDK with the plugins for your chosen providers:

# Install LiveKit Agents and plugins
pip install "livekit-agents[deepgram,openai,elevenlabs,silero]~=1.0"

A complete voice agent that answers questions using GPT-4o and speaks responses with ElevenLabs:

import asyncio
from livekit import agents
from livekit.agents import AutoSubscribe, JobContext, WorkerOptions, cli
from livekit.agents.voice import Agent, AgentSession
from livekit.plugins import deepgram, openai, elevenlabs, silero

async def entrypoint(ctx: JobContext):
    await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY)

    session = AgentSession(
        vad=silero.VAD.load(),                       # Voice Activity Detection
        stt=deepgram.STT(model="nova-2"),            # Speech-to-Text
        llm=openai.LLM(model="gpt-4o-mini"),        # Language Model
        tts=elevenlabs.TTS(voice="Rachel"),          # Text-to-Speech
    )

    await session.start(
        ctx.room,
        agent=Agent(
            instructions="""You are a friendly customer service agent for CelloIP Technologies.
            Help callers with VoIP development enquiries, pricing, and project scope.
            Keep responses concise — under 3 sentences for voice delivery.
            If the caller wants to speak to a human, say you will transfer them.""",
        ),
    )

if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))

3. STT / LLM / TTS Provider Comparison

LiveKit Agents supports any combination of providers. Choose based on latency, quality, cost, and data residency requirements:

4. Key Agent Capabilities

5. Function / Tool Calling — CRM + SIP Transfer

Agents become truly useful when they can act on information — looking up customer records, booking appointments, or transferring calls. The @function_tool decorator registers a Python function that the LLM can call:

# Add function/tool calling — CRM lookup example
from livekit.agents import function_tool
from livekit.agents.voice import Agent, AgentSession

@function_tool
async def lookup_account(phone_number: str) -> str:
    """Look up customer account by phone number."""
    # Replace with real CRM API call
    customer = await crm_api.get_customer(phone=phone_number)
    if customer:
        return f"Found account: {customer.name}, Plan: {customer.plan}"
    return "No account found for this number."

@function_tool
async def transfer_to_human(department: str) -> str:
    """Transfer caller to a human agent in the specified department."""
    # Trigger SIP REFER transfer back to Asterisk
    await sip_client.transfer_participant(
        room_name=ctx.room.name,
        participant_identity="phone_caller",
        transfer_to=f"sip:+1800{department}@asterisk.company.com"
    )
    return f"Transferring to {department} team now."

# Pass tools to agent
session = AgentSession(
    vad=silero.VAD.load(),
    stt=deepgram.STT(model="nova-2"),
    llm=openai.LLM(model="gpt-4o"),
    tts=elevenlabs.TTS(voice="Rachel"),
    tools=[lookup_account, transfer_to_human],
)

6. Handling Inbound SIP / PSTN Calls

To accept real phone calls, configure a SIP inbound trunk in LiveKit and create dispatch rules that route incoming calls to your agent workers:

7. Self-Hosted Stack for GDPR / HIPAA Compliance

All LiveKit components are open source and self-hostable. For EU data residency or healthcare deployments, CelloIP recommends a fully on-premise AI stack: