Pipecat + Asterisk / FreeSWITCH Integration: Real-Time AI Voice Agent Guide

Quick Answer

Pipecat is an open-source Python framework for orchestrating real-time, multimodal AI conversation pipelines, but it has no built-in SIP or PSTN transport. To use it for phone calls, terminate the call normally on Asterisk or FreeSWITCH, then stream the call's raw audio to Pipecat over a TCP socket using AudioSocket (Asterisk's built-in dialplan application for exactly this purpose). Pipecat treats the socket as a custom transport, running speech-to-text, an LLM, and text-to-speech against the live audio stream, and writing synthesized audio back down the same connection.

What Is Pipecat?

Pipecat is an open-source Python framework, backed by Daily, for building real-time voice and multimodal AI agents. It models a conversation as a pipeline of composable frame processors — audio input, voice activity detection, speech-to-text, LLM context management, text-to-speech, and output — with built-in handling for interruptions (barge-in), turn-taking, and low-latency streaming. Pipecat reached its v1.0 release in April 2026, and its plugin ecosystem now covers most major STT providers (Deepgram, AssemblyAI, Whisper), LLMs (OpenAI, Anthropic, Gemini), and TTS engines (ElevenLabs, Cartesia, PlayHT).

Pipecat's native transports are built around WebRTC (via Daily's own infrastructure or LiveKit) and browser/WebSocket use cases — it was not designed with SIP telephony as a first-class citizen. For teams that want Pipecat's pipeline flexibility but need it to answer real phone numbers, a telephony bridge is required.

Why Bridge Pipecat to Asterisk or FreeSWITCH Instead of Using It Standalone?

Managed voice AI platforms like VAPI, Retell, and Bland handle the telephony layer for you, but charge a per-minute platform fee on top of the underlying model costs, and give you less control over call routing, SIP trunk selection, and compliance-sensitive call handling. Bridging Pipecat directly to Asterisk or FreeSWITCH means:

• Full control over SIP trunk selection, failover, and carrier routing — the same infrastructure you'd use for any production PBX
• No per-minute platform fee — you pay only for STT/LLM/TTS API usage plus your own server hosting
• Existing Asterisk/FreeSWITCH dialplan logic (IVR fallback, call transfer to a human agent, CDR/billing integration) stays in the telephony layer where it already works
• Data stays on infrastructure you control — relevant for HIPAA, financial services, or any compliance-sensitive deployment

Architecture: Call Flow From Phone to Pipecat and Back

A caller dials in over a standard SIP trunk, which Asterisk terminates exactly as it would for any inbound call. Instead of routing to a human extension, the dialplan invokes the AudioSocket() application, which opens a plain TCP connection to a local bridge process and streams the call's audio as 8kHz, 16-bit signed linear PCM in 320-byte frames (20ms of audio per frame) — both directions, simultaneously. That bridge process is where Pipecat lives: it wraps the TCP socket in a custom Pipecat transport, feeding incoming frames into the STT → LLM → TTS pipeline and writing synthesized speech frames back onto the same socket, which Asterisk then plays to the caller in real time.

Runnable Bridge Code: AudioSocket to Pipecat

Dialplan snippet to invoke AudioSocket from Asterisk, followed by a minimal Python bridge that wraps the socket for a Pipecat pipeline:

extensions.conf

[from-trunk]
exten => _X.,1,NoOp(Inbound call to AI voice agent)
 same => n,Answer()
 same => n,AudioSocket(${UNIQUEID},127.0.0.1:9999)
 same => n,Hangup()

bridge.py — AudioSocket ↔ Pipecat transport

import asyncio
import struct

FRAME_BYTES = 320  # 20ms of 8kHz 16-bit mono PCM

async def handle_call(reader: asyncio.StreamReader,
                       writer: asyncio.StreamWriter):
    # AudioSocket header: 1-byte type + 2-byte length + payload
    call_id = None
    while True:
        header = await reader.readexactly(3)
        kind, length = header[0], struct.unpack(">H", header[1:3])[0]
        payload = await reader.readexactly(length) if length else b""

        if kind == 0x01:          # "hello" — carries the call UUID
            call_id = payload
        elif kind == 0x10:        # audio frame from the caller
            await pipecat_pipeline.push_audio_in(payload)  # -> STT/LLM/TTS
        elif kind == 0x00:        # hangup
            break

    # Pipecat pushes synthesized TTS frames back out:
    # writer.write(b"\x10" + struct.pack(">H", len(pcm_chunk)) + pcm_chunk)

async def main():
    server = await asyncio.start_server(handle_call, "127.0.0.1", 9999)
    async with server:
        await server.serve_forever()

asyncio.run(main())

In production, replace the direct pipecat_pipeline.push_audio_in() call with Pipecat's custom-transport interface so the pipeline's VAD, interruption handling, and frame-processor chain run exactly as they would over a WebRTC transport.

The FreeSWITCH Path: mod_audio_fork

A Pipecat FreeSWITCH integration follows the same shape as the Asterisk path but uses a different module. FreeSWITCH doesn't ship AudioSocket, but mod_audio_fork achieves the equivalent result: it forks a call's audio to an external WebSocket endpoint in real time, bidirectionally, without interrupting the underlying call leg. Where Asterisk gives you a raw TCP socket with a 3-byte frame header, FreeSWITCH's fork module speaks WebSocket, so the bridge code on the Pipecat side swaps a TCP server for a WebSocket server — everything downstream of that (the STT/LLM/TTS pipeline itself) is identical.

<!-- dialplan.xml -->
<extension name="ai-voice-agent">
  <condition field="destination_number" expression="^(.*)$">
    <action application="answer"/>
    <action application="audio_fork"
            data="wss://127.0.0.1:9000 mono 16000"/>
  </condition>
</extension>

One practical difference worth planning around: mod_audio_fork streams at whatever sample rate you configure (commonly 8kHz or 16kHz), while Asterisk's AudioSocket is fixed at 8kHz. If your STT provider performs meaningfully better at 16kHz — Deepgram and AssemblyAI both do — a FreeSWITCH deployment gets a small but real accuracy advantage over Asterisk for the same pipeline, at the cost of double the bandwidth per call leg.

Choosing STT, LLM, and TTS Providers

Pipecat is provider-agnostic, which means the pipeline's actual latency and quality come almost entirely from which STT, LLM, and TTS services you plug in — not from Pipecat itself. For a telephony bridge specifically, three things matter more than they do for a browser-based voice agent: first-byte latency (the phone channel has no visual feedback to mask a pause), 8kHz/16kHz audio compatibility (most STT providers are tuned on 16kHz+ audio and lose accuracy at telephony bandwidth), and streaming support (batch-only APIs are a non-starter for a live call).

• Deepgram Nova-2 — our default STT choice for phone audio: streaming, tuned for 8kHz telephony, ~200–300ms first-partial latency
• OpenAI GPT-4o or Anthropic Claude Haiku — GPT-4o for tool-calling-heavy flows (CRM lookups, scheduling), Claude Haiku when you need lower per-token cost at similar quality for simpler scripted flows
• ElevenLabs Turbo or Cartesia Sonic — both stream synthesized audio in under 200ms to first byte, which is the TTS latency floor that makes barge-in feel natural rather than sluggish

A detail that catches teams off guard the first time: telephony audio is 8kHz mono, and several STT providers quietly perform worse below their tested sample rate without erroring — you get correct-looking transcripts with a higher word error rate on accented speech, background noise, or crosstalk, and it's easy to miss in a demo with a quiet room and a clear voice. Test with real call recordings, not a clean microphone, before trusting accuracy numbers from a provider's marketing page.

Handling Barge-In (Interruptions)

Because AudioSocket is bidirectional and full-duplex, the caller's audio keeps streaming in even while your TTS is playing out. Pipecat's built-in voice activity detection can flag caller speech mid-response; when that happens, the bridge should immediately stop writing further TTS frames to the socket and flush any buffered audio, so the caller doesn't hear a queued response continue after they've started talking. This is the single most common defect in first-pass AudioSocket bridges — teams that skip explicit flush-on-interrupt logic ship agents that talk over the caller.

Production Deployment Checklist

A working demo and a production Pipecat Asterisk or FreeSWITCH deployment are different projects. Before putting real calls through the bridge, verify each of these:

• Reconnection handling — if the TCP/WebSocket bridge process restarts mid-call, does the call hang up cleanly or hang silently? Asterisk and FreeSWITCH both need a dialplan fallback (transfer to human queue) if the bridge connection drops.
• Concurrent call limits — size the bridge process's connection pool and the telephony server's channel limits together; a bridge that accepts unlimited connections but chokes past 50 concurrent STT/LLM calls will degrade silently under load, not fail loudly.
• API rate limits on STT/LLM/TTS providers — most providers cap concurrent streaming connections per account tier; hitting that ceiling mid-traffic-spike looks identical to a bridge bug from the caller's side.
• Timeout handling for a silent caller — decide explicitly how long the agent waits before re-prompting or ending the call, rather than leaving Pipecat's default VAD timeout in place untested.
• Call recording and consent — if you record for QA or compliance, the recording hook lives in the telephony layer (Asterisk MixMonitor or FreeSWITCH record_session), not in Pipecat, and needs its own retention and access-control policy.
• Graceful degradation — define what happens if the LLM call times out or errors mid-conversation: a scripted apology-and-transfer path is far better than a caller hearing dead air.

Monitoring and Observability

Because the telephony layer (Asterisk/FreeSWITCH), the bridge process, and the AI pipeline are three separate systems, a production deployment needs visibility into each independently, not just an overall "is the number answering" check. Track concurrent AudioSocket/mod_audio_fork connections against configured limits, STT/LLM/TTS provider latency per call (not just averages — p95 and p99 are what tell you when a provider is degrading), bridge process memory and file-descriptor usage (a common leak source when connections aren't closed cleanly on hangup), and end-to-end response latency from caller-stops-speaking to agent-starts-speaking, which is the single number that best predicts whether callers perceive the agent as natural or laggy. Asterisk exposes channel and call state via AMI/ARI events; FreeSWITCH exposes the same via ESL — both integrate cleanly with Prometheus exporters if you're already running Grafana for other infrastructure.

Common Pitfalls

Beyond barge-in, the mistakes we see most often building Pipecat Asterisk and Pipecat FreeSWITCH bridges for clients:

• Blocking the event loop with synchronous provider SDKs — several STT/TTS Python SDKs default to synchronous HTTP clients; used naively inside an asyncio bridge, they stall audio processing for every concurrent call, not just the one making the request.
• Ignoring codec mismatches — AudioSocket and mod_audio_fork both expect raw linear PCM, not the G.711 µ-law/A-law that arrives from the SIP trunk; the telephony server transcodes this automatically, but a custom dialplan that skips the standard codec negotiation step will hand your bridge garbled audio.
• Testing only with clean, quiet audio — real callers have background noise, phone-hold music bleed-through, and regional accents; a pipeline tuned entirely on a quiet office microphone will show a materially higher word-error rate in production.
• No fallback for provider outages — treating VAPI/Retell-grade uptime as a given for a self-hosted stack that has three additional external dependencies (STT, LLM, TTS) each with their own SLA and occasional incidents.

Cost: Self-Hosted Pipecat vs VAPI/Retell

The self-hosted path is cheaper per minute but shifts telephony reliability, scaling, and monitoring onto your own team — the right trade-off once call volume is high enough to justify the operational investment.

Run the math at a realistic volume: 20,000 minutes/month is a modest but real contact-center workload. At $0.05/min all-in, self-hosted Pipecat costs roughly $1,000/month in API usage plus $150–$300/month for the bridge server and telephony infrastructure — call it $1,300/month total. The same volume on Retell AI at $0.10/min all-in runs $2,000/month with zero infrastructure to operate. The self-hosted path saves roughly $700/month at this volume, and the gap widens linearly as volume grows, which is exactly why the decision should be driven by expected call volume and available ops capacity, not by which approach looks simpler on day one.

FAQ