LiveKitGemini LiveVoice AgentRealtimeSIPPython

LiveKit + Gemini Live: How to Build a Real-Time Voice Agent

Gemini Live lets an AI agent hear and speak in one model, and LiveKit carries the conversation from a browser, a mobile app or a phone line. Together they make a voice agent with fewer moving parts than a speech-to-text, LLM and text-to-speech pipeline. This guide covers when realtime is the right choice, setup, working agent code, tools, phone calls through LiveKit SIP, turn-taking, current model options, costs from Google's price list, and what to check before production.

Kaushik ParmarFounder & VoIP Architect ยท 12+ years in telecom and VoIP 14 min read
1
Model hears and speaks
$0.005
Per min audio in (Gemini 3.8 Live)
$0.018
Per min audio out (Gemini 3.8 Live)
6
Steps from setup to phone calls

Quick answer

How do you build a voice agent with LiveKit and Gemini Live?

To build a voice agent with LiveKit and Gemini Live, install livekit-agents with the Google plugin, create an AgentSession whose LLM is google.realtime.RealtimeModel, and run it as a LiveKit Agents worker. Gemini Live hears the caller's audio and speaks directly, without separate speech-to-text and text-to-speech. LiveKit carries the call from browsers, apps or phone lines through LiveKit SIP, and your worker adds tools such as CRM lookups and booking.

What you are building

LiveKit and Gemini Live voice agent architecture: phone callers arrive through a SIP trunk and LiveKit SIP, web and app users join over WebRTC, both reach a LiveKit room; an agent worker streams audio to the Gemini Live API and calls CRM and booking tools.
LiveKit carries the call; Gemini Live listens and speaks; your worker owns tools, rules and logging.

A voice agent built on LiveKit and Gemini Live has three parts. LiveKit is the real-time transport: a WebRTC media server (SFU) where callers and the agent meet in a room, with SDKs for web and mobile and LiveKit SIP for phone calls. The agent worker is your Python program, built with the LiveKit Agents framework, which joins the room, applies your instructions and tools, and logs what happens. Gemini Live is Google's real-time model: the worker streams the caller's audio to it and plays back the audio it returns.

The difference from a classic voice agent is that there is no separate speech-to-text or text-to-speech step. A native audio model hears tone and timing directly and speaks with natural pacing, which tends to make conversations feel less robotic and removes two network hops. The trade-off is less control: you choose from the model's voices, and you depend on one provider for listening, thinking and speaking.

If you are new to LiveKit voice agents, our LiveKit voice agents guide covers the pipeline approach and the framework basics; this guide focuses on the Gemini realtime path.

Realtime vs pipeline: which to choose

Comparison of voice agent designs: a pipeline of speech-to-text, LLM and text-to-speech; a realtime speech-to-speech model; and a half-cascade where the realtime model understands speech and your own text-to-speech speaks.
Three ways to wire speech into an AI voice agent.
FactorRealtime (Gemini Live native audio)Pipeline (STT โ†’ LLM โ†’ TTS)Half-cascade
Moving partsOne modelThree providersTwo
Conversation feelNatural pacing and interruptionsGood, with careful tuningNatural listening, chosen voice
Voice choiceThe model's built-in voicesAny TTS voice, including customAny TTS voice
Accent-specific speech recognitionNot separately tunableChoose the best STT per languageNot separately tunable
Transcripts for loggingFrom the model's transcription outputFrom the STT stepFrom the model
Vendor dependenceOne provider for everythingSpread across providersTwo providers
Best forFast, natural assistants, receptionists, supportBrand voice, regulated scripts, tricky accentsNatural listening with a brand voice

A practical rule: prototype on realtime because it is quick to build and sounds good, then test it against your hardest calls, such as noisy lines, strong accents, long account numbers and callers who interrupt. If it handles them, keep it. If not, the same LiveKit worker can switch to a pipeline with different speech providers without changing your tools or telephony. Our voice AI latency and barge-in guide explains how to measure the difference.

Setup

  1. 1. Create a LiveKit project

    Use LiveKit Cloud or a self-hosted LiveKit server, and note the URL, API key and secret.

  2. 2. Get Gemini credentials

    Create a Gemini API key, or set up a Google Cloud project and service account for Vertex AI.

  3. 3. Install the agent framework

    Install livekit-agents with the Google extra in a Python environment.

  4. 4. Write the agent

    Create an Agent with instructions and tools and an AgentSession that uses google.realtime.RealtimeModel.

  5. 5. Run and test in a browser

    Start the worker in development mode and talk to it from the LiveKit Agents playground or your own web client.

  6. 6. Connect phone numbers

    Create a LiveKit SIP inbound trunk for your numbers and a dispatch rule that sends calls to the agent.

# Versions current in LiveKit docs on 19 Nov 2026
uv add "livekit-agents[google]~=1.8"        # or: pip install "livekit-agents[google]~=1.8"

# .env
LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=...
LIVEKIT_API_SECRET=...

# Gemini Developer API
GOOGLE_API_KEY=...

# or Vertex AI instead
# GOOGLE_GENAI_USE_VERTEXAI=true
# GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
# GOOGLE_CLOUD_PROJECT=your-project
# GOOGLE_CLOUD_LOCATION=us-central1

Choose between the Gemini Developer API and Vertex AI early. The Developer API is the quickest start. Vertex AI fits teams that already run on Google Cloud and need its project-level controls, regions and billing. Model availability and plugin version requirements differ between the two, so check both before you commit.

A minimal Gemini Live agent

The agent below follows the structure in LiveKit's documentation: an Agent with instructions, an AgentSession whose LLM is google.realtime.RealtimeModel, and a worker entrypoint. The agent_name lets LiveKit dispatch this specific agent to phone calls later.

# agent.py
from dotenv import load_dotenv
from livekit import agents
from livekit.agents import Agent, AgentSession
from livekit.plugins import google

load_dotenv()

INSTRUCTIONS = """You are the receptionist for Northwind Dental.
Speak in short, friendly sentences. Confirm names, dates and phone numbers
by repeating them back. If you cannot help, offer to transfer to a person."""

class Receptionist(Agent):
    def __init__(self) -> None:
        super().__init__(instructions=INSTRUCTIONS)

async def entrypoint(ctx: agents.JobContext):
    session = AgentSession(
        llm=google.realtime.RealtimeModel(
            voice="Puck",               # plugin default voice
            temperature=0.6,
        ),
    )
    await session.start(room=ctx.room, agent=Receptionist())
    await ctx.connect()
    await session.generate_reply(instructions="Greet the caller and ask how you can help.")

if __name__ == "__main__":
    agents.cli.run_app(agents.WorkerOptions(entrypoint_fnc=entrypoint, agent_name="gemini-receptionist"))

# Run locally:  python agent.py dev

Keep instructions short and concrete. Realtime models follow spoken-style instructions well: say how to speak, what to confirm, and what to do when unsure. Long policy documents belong in tools or retrieval, not in the system prompt. If you rely on generate_reply together with function tools on the Gemini API, LiveKit's docs note that reply control requires the BLOCKING tool behaviour, covered next.

Tools: letting the agent do real work

A voice agent becomes useful when it can look things up and take actions. In LiveKit Agents, you define tools as methods on the agent; the framework describes them to Gemini, and Gemini calls them during the conversation.

from livekit.agents import Agent, RunContext, function_tool

class Receptionist(Agent):
    def __init__(self) -> None:
        super().__init__(instructions=INSTRUCTIONS)

    @function_tool()
    async def find_next_appointment(self, context: RunContext, phone_number: str) -> str:
        """Find the caller's next appointment using their phone number in E.164 format."""
        appt = await clinic_api.next_appointment(phone_number)
        if not appt:
            return "No upcoming appointment found."
        return f"{appt.patient_first_name} on {appt.date:%A %d %B} at {appt.time:%H:%M} with {appt.dentist}."

    @function_tool()
    async def book_appointment(self, context: RunContext, phone_number: str, date: str, time: str) -> str:
        """Book an appointment. Only call after the caller has confirmed date and time."""
        result = await clinic_api.book(phone_number, date, time)
        return "Booked." if result.ok else f"Could not book: {result.reason}"

# Control whether Gemini keeps talking while a tool runs:
# google.realtime.RealtimeModel(..., tool_behavior="BLOCKING")   # or "NON_BLOCKING"
  • Give each tool one job and a clear docstring; the model reads it
  • Validate every argument on the server; never trust the model's input
  • Ask for confirmation in the instructions before any tool that changes data
  • Return short, speakable results rather than raw JSON
  • Log every tool call with the call ID for auditing
  • Keep CRM credentials in the worker, never in the prompt

Tool behaviour matters for conversation feel. With NON_BLOCKING, the model can keep talking ("let me check that for you") while a slow lookup runs. With BLOCKING, it waits for the result. LiveKit's documentation notes that some Gemini models do not support asynchronous function calling, so test the model you choose. Our LiveKit MCP agent integration guide shows how to expose tools through MCP instead.

Phone calls through LiveKit SIP

To answer phone calls, point your phone numbers at LiveKit SIP, then tell LiveKit what to do with each call. An inbound trunk lists your numbers and allowed sources; a dispatch rule puts each call in its own room and dispatches your named agent into it.

// inbound-trunk.json
{
  "trunk": {
    "name": "Main numbers",
    "numbers": ["+15105550123"]
  }
}

// dispatch-rule.json: one room per call, with the Gemini agent
{
  "dispatch_rule": {
    "rule": { "dispatchRuleIndividual": { "roomPrefix": "call-" } },
    "room_config": { "agents": [{ "agent_name": "gemini-receptionist" }] }
  }
}

# Create them with the LiveKit CLI
lk sip inbound create inbound-trunk.json
lk sip dispatch create dispatch-rule.json

Your SIP provider then routes the numbers to your LiveKit SIP endpoint. Check LiveKit's SIP documentation for the current field names and authentication options before going live, because the SIP API evolves with each release. If your numbers already live on an Asterisk or FreeSWITCH PBX, you can keep them there and route selected calls to LiveKit over SIP; our LiveKit with Asterisk and FreeSWITCH guide and adding an AI voice agent to an existing phone system cover that design.

Turn-taking and interruptions

Knowing when the caller has finished speaking is the hardest part of any voice agent. Gemini Live has built-in voice activity detection that decides when to respond and handles barge-in when the caller talks over the agent. That is on by default and works well for most conversations.

LiveKit also has its own turn detector model, which looks at the words as well as the silence to judge whether a caller has really finished. LiveKit's documentation explains how to turn off Gemini's automatic activity detection and use LiveKit's turn handling instead. Consider it if callers are often cut off mid-sentence, for example while reading out a long account number or pausing to think.

ProblemLikely causeWhat to try
Agent cuts in while the caller pausesSilence-based end-of-turn too eagerLiveKit's turn detector; instruct the agent to wait for numbers
Agent keeps talking when interruptedInterruptions not reaching the model quicklyCheck audio path and echo cancellation on the client
Long pause before repliesSlow tools or network distance to the modelNON_BLOCKING tools with filler phrases; host the worker near the model region
Agent mishears numbersLine quality, accentsAsk the caller to confirm; read back; consider a pipeline with tuned STT

Models and options

Gemini Live models change frequently, so treat this table as a snapshot of LiveKit's documentation on 19 November 2026 and check it again before you build.

SettingWhat it doesNotes from LiveKit's docs
modelWhich Gemini Live model to useDefaults: gemini-2.5-flash-native-audio-preview-12-2025 (Gemini API), gemini-live-2.5-flash-native-audio (Vertex AI). gemini-3.8-live on Vertex needs plugin 1.8.5+ (Python)
voiceThe built-in voiceDefault Puck
modalitiesAUDIO for speech-to-speech, TEXT for half-cascadeHalf-cascade works only with non-native-audio models
instructionsSystem instructionsCan also be set on the Agent
tool_behaviorBLOCKING or NON_BLOCKING tool callsPython defaults to NON_BLOCKING on Gemini 3.8 Live
thinking_configReasoning settingsModel-specific; Gemini 3.1 uses thinkingLevel
enable_affective_dialog, proactivityEmotion-aware replies; proactive audioNative audio models only; not supported on Gemini 3.1
vertexai, project, locationUse Vertex AI instead of the Developer APIOr set GOOGLE_GENAI_USE_VERTEXAI and related variables

If the plugin does not recognise a model name, it logs a warning and still tries to connect, and the connection fails if the API does not serve that model. Pin your plugin version and model name together in production, and upgrade them together after testing.

Vision: agents that can see

Gemini Live accepts video frames as well as audio, and LiveKit publishes a recipe for a Gemini realtime agent with live vision. A user can point a phone camera at a product, a meter or a document and ask about it, and the agent answers by voice. LiveKit documents video input for its Python framework.

Field support

A technician shows equipment; the agent identifies the model and reads the error light.

Insurance and claims

A customer shows damage while describing what happened; the agent fills the claim form.

Retail help

A shopper shows a product; the agent finds the matching item and its stock.

Video input costs more than audio and raises privacy questions, so send frames only when needed, at low resolution, and tell users when the camera is being analysed.

Cost per minute

Google's price list for Gemini 3.8 Live, checked on 19 November 2026, gives paid-tier prices of $3.00 per million audio input tokens, about $0.005 per minute of audio in, and $12.00 per million audio output tokens, about $0.018 per minute of audio out. Text input and output tokens are billed separately.

These are model costs only. A full per-minute cost also includes LiveKit (cloud or your own servers), phone numbers and carrier minutes, the worker's hosting, tool APIs and any recording or transcription storage. Our AI voice agent cost guide and self-hosted vs managed voice AI guide cover total cost, and prices change often, so check each provider's pricing page before you budget.

Gemini Live vs OpenAI realtime

FactorGemini Live (via LiveKit plugin)OpenAI realtime
LiveKit Agents supportYes, google.realtime.RealtimeModelYes, via the OpenAI plugin
Direct SIP from the providerUse LiveKit SIP or your PBXOpenAI offers native SIP; see our guide
Listed price (19 Nov 2026)Gemini 3.8 Live: about $0.005/min audio in, $0.018/min audio outgpt-live-1: $0.05/min, backend model billed separately
Video inputYes (Python in LiveKit)Check current model support
Cloud platformGemini API or Vertex AI on Google CloudOpenAI API or Azure OpenAI

Both are strong choices, and LiveKit makes switching between them a change of a few lines. Choose on what you can test: voice quality in your languages, behaviour on your hardest calls, tool reliability and data-processing terms. For the OpenAI side, see our OpenAI Realtime native SIP vs bridge guide.

Production checklist

  • Pin livekit-agents, the Google plugin and the Gemini model name together
  • Test on real phone audio, not just a laptop microphone
  • Run your hardest calls: noise, accents, long numbers, interruptions
  • Confirm and read back names, dates and numbers in the instructions
  • Validate every tool argument server-side and log every tool call
  • Add a transfer-to-human path and test it on every number
  • Record or transcribe calls only with the consent your region requires
  • Monitor latency, tool errors and call outcomes per day
  • Set per-call and per-day spending limits on the model account
  • Re-test after any model or plugin upgrade

Our voice agent testing and monitoring guide covers how to build a test suite of recorded calls and what to watch in production.

A real example: a contact centre voice agent

CelloIP built an AI voice agent for a North American contact centre on FreeSWITCH and LiveKit Agents with hosted speech and language APIs. It handles inbound and outbound calls in English and Spanish, works with the CRM, the booking system and the helpdesk, and hands calls to people with context. The same LiveKit architecture supports either a pipeline or a realtime model such as Gemini Live, so the model can change as providers improve without rebuilding telephony or integrations.

CelloIP project outcome (client kept private)

Inbound and outbound calls in English and Spanish, integrated with CRM, booking and helpdesk, built on FreeSWITCH and LiveKit Agents and maintained by CelloIP.

Frequently asked questions

What is the Gemini Live API?

It is Google's low-latency API for real-time conversations with Gemini over a bidirectional stream. Native audio models take audio in and produce audio out in one model, so the agent does not need separate speech-to-text and text-to-speech services, and it can handle interruptions and natural turn-taking.

Does LiveKit support Gemini Live?

Yes. The LiveKit Agents Google plugin provides google.realtime.RealtimeModel, which you pass as the llm of an AgentSession. It works with the Gemini Developer API using GOOGLE_API_KEY or with Vertex AI using a service account. Older examples use google.beta.realtime, which has been replaced.

Which Gemini model does the LiveKit plugin use by default?

According to LiveKit's documentation at the time of writing, the default on the Gemini API is gemini-2.5-flash-native-audio-preview-12-2025, and on Vertex AI it is gemini-live-2.5-flash-native-audio. Newer models such as gemini-3.8-live can be selected with the model parameter; on Vertex AI that needs livekit-plugins-google 1.8.5 or later.

Can a Gemini Live agent answer phone calls?

Yes, through LiveKit SIP. You create an inbound SIP trunk for your phone numbers and a dispatch rule that puts each call into a room and sends your agent to it. The caller hears Gemini's voice through the same room the agent uses for web and app callers.

How much does Gemini Live cost per minute?

Google lists Gemini 3.8 Live at about $0.005 per minute of audio input and $0.018 per minute of audio output on its paid tier, as of 19 November 2026. A real conversation also includes text and context tokens, plus LiveKit, telephony and hosting costs, so measure a pilot before you budget.

Should I use Gemini Live or a speech-to-text, LLM and text-to-speech pipeline?

Use Gemini Live when natural conversation and low latency matter most and its voices and languages suit you. Use a pipeline when you need a specific brand voice, a particular speech-to-text model for your accents, or full control over each step. A half-cascade, Gemini Live understanding speech with your own text-to-speech, is a middle path, though LiveKit notes it only works with non-native-audio models.

Can Gemini Live agents call functions?

Yes. Function tools defined in your LiveKit agent are passed to Gemini, which can call them during the conversation. The tool_behavior setting controls whether the model keeps talking while a tool runs; on the Gemini API, reply control needs BLOCKING, and some models do not support asynchronous function calling.

What does a production voice agent on LiveKit cost to build?

As a guide, a managed-platform MVP for one use case is typically $8,000 to $20,000 over 4 to 6 weeks, and a custom production agent on a self-hosted framework such as LiveKit is typically $25,000 to $60,000 over 8 to 16 weeks, depending on telephony, integrations and languages.

Work with CelloIP

Building a voice agent on LiveKit?

CelloIP builds production voice agents on LiveKit with Gemini, OpenAI or pipeline models, connected to your phone numbers, PBX and business systems, in any language your customers speak. We reply within one business day, and you own the code.