Asterisk AudioSocket: Real-Time AI Voice Agent Streaming Guide

The AudioSocket Frame Format

Every AudioSocket message is a simple 3-part frame: a 1-byte type field (0x00 hangup, 0x01 UUID/hello, 0x10 audio payload, 0x03 error), a 2-byte big-endian length field, and a payload of up to 320 bytes. Audio frames carry 320 bytes of 8kHz, 16-bit signed linear PCM — exactly 20 milliseconds of mono audio — sent continuously in both directions for the duration of the call.

Because the connection is a single persistent TCP socket carrying both directions simultaneously, your application can read incoming caller audio and write synthesized response audio at the same time — there is no request/response round-trip the way there is with AGI.

Why AudioSocket Beats AGI for Real-Time AI

AGI (Asterisk Gateway Interface) is a synchronous, command-based protocol: your script issues a command like STREAM FILE and blocks until Asterisk responds. This model works well for traditional IVR logic — play a prompt, collect DTMF, branch — but it was never designed to stream continuous raw audio for an external process to analyze in real time. AudioSocket solves exactly that gap: it hands your application the raw media stream directly, with no dialplan command round-trip in the loop, which is what makes sub-second AI voice agent response times achievable.

• Non-blocking: audio streams continuously; your process reads and writes independently of Asterisk's dialplan execution
• Full-duplex: caller audio in and synthesized audio out flow simultaneously on the same socket — required for natural barge-in handling
• Minimal protocol overhead: a 3-byte header per frame versus AGI's text command/response parsing
• Language-agnostic: any TCP-capable language can implement an AudioSocket client — Python, Node.js, Go, Rust

Dialplan Configuration

[ai-voice-agent]
exten => _X.,1,NoOp(Routing to AI voice agent)
 same => n,Answer()
 same => n,Set(UUID=${SHELL(uuidgen)})
 same => n,AudioSocket(${UUID},10.0.0.5:9000)
 same => n,Hangup()

Python Frame Reader/Writer

import asyncio, struct

TYPE_HANGUP, TYPE_UUID, TYPE_AUDIO, TYPE_ERROR = 0x00, 0x01, 0x10, 0x03

async def read_frame(reader: asyncio.StreamReader):
    header = await reader.readexactly(3)
    kind, length = header[0], struct.unpack(">H", header[1:3])[0]
    payload = await reader.readexactly(length) if length else b""
    return kind, payload

def write_audio_frame(writer: asyncio.StreamWriter, pcm: bytes):
    for i in range(0, len(pcm), 320):
        chunk = pcm[i:i + 320]
        writer.write(bytes([TYPE_AUDIO]) + struct.pack(">H", len(chunk)) + chunk)

async def handle_call(reader, writer):
    while True:
        kind, payload = await read_frame(reader)
        if kind == TYPE_HANGUP:
            break
        if kind == TYPE_AUDIO:
            # payload -> STT -> LLM -> TTS, then:
            # write_audio_frame(writer, synthesized_pcm)
            pass
    writer.close()

Building a Production AudioSocket Server

The frame reader/writer above is correct but incomplete for production use — it has no handling for a client that disconnects mid-frame, no timeout on a caller who never sends the expected UUID hello frame, and no isolation between concurrent calls sharing the same asyncio event loop. A production Asterisk AudioSocket server needs three additional things: per-connection timeout enforcement so a hung TCP connection doesn't leak a file descriptor forever, exception handling around each call's frame loop so one malformed frame doesn't crash the entire server process, and a connection registry keyed by call UUID so you can look up, log, and forcibly terminate a specific call's bridge session if it needs to be killed from outside the frame loop — useful for admin tooling and for graceful shutdowns during deployments.

active_calls: dict[str, asyncio.StreamWriter] = {}

async def handle_call(reader, writer):
    call_id = None
    try:
        while True:
            kind, payload = await asyncio.wait_for(
                read_frame(reader), timeout=30.0
            )
            if kind == TYPE_UUID:
                call_id = payload.decode()
                active_calls[call_id] = writer
            elif kind == TYPE_HANGUP:
                break
            elif kind == TYPE_AUDIO:
                await pipeline_for(call_id).push_audio_in(payload)
    except (asyncio.TimeoutError, ConnectionResetError) as e:
        log.warning(f"call {call_id} bridge error: {e}")
    finally:
        active_calls.pop(call_id, None)
        writer.close()

Why the UUID Hello Frame Matters More Than It Looks

It's easy to treat the type-0x01 UUID frame as a formality and move straight to processing audio, but it's the only piece of call-identifying information AudioSocket gives you — everything else in the frame stream is opaque PCM bytes. In a system handling more than a handful of concurrent calls, that UUID is what lets your bridge correlate an incoming connection with the right customer record, the right conversation history if a caller re-dials, and the right billing/CDR entry once the call ends. Asterisk generates this UUID from the dialplan (typically via ${UNIQUEID} or a custom-generated value passed as the AudioSocket argument), so it's also the join key if you need to cross-reference AudioSocket-side logs against Asterisk's own CDR or AMI event log when debugging a specific call after the fact. Systems that skip capturing and indexing this value on the bridge side consistently regret it the first time they need to investigate a single problematic call out of thousands.

Security Considerations

AudioSocket has no built-in authentication or encryption — it is a bare TCP protocol, designed to run on a trusted internal network between Asterisk and your bridge process. This is fine when both sides sit in the same private VPC or Docker network, which is how CelloIP deploys it by default. It becomes a real risk if the bridge process is ever reachable from outside that trusted boundary, since anyone who can reach the port can open a connection and impersonate a call, or intercept live call audio in plaintext.

• Bind the AudioSocket listener to a private interface or Unix socket, never 0.0.0.0 on a public-facing host
• If Asterisk and the bridge run on different hosts, put the link inside a VPN or private VPC peering — do not route raw AudioSocket traffic over the public internet
• Validate the UUID hello frame against Asterisk's own channel/call records before trusting a connection, so a stray or malicious connection can't inject fake audio into your pipeline
• Log connection source IPs and alert on any connection attempt from outside the expected Asterisk host — the protocol gives you no other signal that something unexpected is happening

Scaling to Hundreds of Concurrent Calls

A single-process asyncio AudioSocket server handles a surprising number of concurrent connections — the protocol itself is lightweight, and most of the CPU cost lives in the STT/LLM/TTS calls, not in frame parsing. In practice, plan capacity around your AI pipeline's concurrency limits rather than AudioSocket's: once you're running more concurrent calls than a single process can service without queueing delay (commonly 50–100 depending on hardware), run multiple bridge processes behind a simple dispatch layer, with the dialplan choosing a target port based on a round-robin or least-connections rule evaluated at call setup. Each bridge process should expose a health-check endpoint reporting its current connection count, so the dispatch layer can route new calls away from an already-saturated instance.

Debugging Common Issues

Three problems account for most of the support requests we see on Asterisk AudioSocket integrations:

• Silence with no error — usually a codec mismatch. Confirm the dialplan answers the call before invoking AudioSocket, and that no earlier dialplan step has already set an incompatible codec on the channel.
• Choppy or garbled audio — almost always a buffering issue where frames are read but not processed at real-time pace; profile whether your STT call is blocking the frame-read loop rather than running concurrently.
• Call hangs up after exactly the timeout period — your bridge is waiting on a frame that never arrives because the UUID hello frame was missed or misparsed; log every frame type received in the first two seconds of a new connection to catch this.

Handling Barge-In

Because AudioSocket carries both directions on one socket, your application always keeps receiving caller audio frames even while it's writing synthesized speech frames out. Run voice activity detection on the incoming stream in parallel with playback; the moment it detects caller speech during your own TTS output, stop writing new audio frames immediately and discard anything still queued. Skipping this step is the most common defect in AudioSocket-based voice agents — without it, the agent keeps talking over an interrupting caller.

AudioSocket in Multi-Tenant Systems

A hosted PBX or contact-center platform serving multiple tenant organizations adds a wrinkle AudioSocket doesn't solve by itself: the protocol has no concept of tenancy, so isolating each tenant's AI agent configuration — which LLM prompt, which voice, which knowledge base — has to happen in your bridge application, keyed off the call UUID or the dialplan context that invoked AudioSocket. The cleanest pattern we've used in production: pass the tenant identifier as part of the AudioSocket connection target (e.g., a per-tenant port range, or a query parameter carried in the dialplan's Set() variables and looked up by the bridge on the UUID hello frame), then have the bridge process load that tenant's pipeline configuration before the first audio frame arrives. Skipping this and hard-coding a single pipeline configuration is the most common reason a proof-of-concept AudioSocket integration can't be sold as a multi-tenant product without a rewrite.

AudioSocket vs RTP and WebRTC Data Channels

It's worth being precise about what AudioSocket is not, since the comparison comes up often. RTP is the protocol that carries media between SIP endpoints during a normal call — AudioSocket doesn't replace RTP, it runs alongside it, receiving a copy of the decoded audio after Asterisk has already handled the SIP/RTP call setup. WebRTC DataChannels, by contrast, are a browser-native peer-to-peer channel with no direct relationship to Asterisk's dialplan at all — they matter if you're building a browser-based voice agent, not a phone-network one. For a PSTN or SIP-trunk-originated call reaching an AI agent, AudioSocket (or FreeSWITCH's mod_audio_fork) is the correct layer; RTP and WebRTC solve adjacent but different problems in the same overall system.

AudioSocket vs AGI vs ARI

FAQ

Summary: When to Reach for Asterisk AudioSocket

Asterisk AudioSocket is the right tool specifically when a call needs to be handed, in real time, to code that lives outside Asterisk — an AI voice agent pipeline, a custom transcription service, or any process that needs raw audio frames rather than dialplan-level control. It is not the right tool for straightforward call routing, IVR menus, or recording, where AGI, ARI, and native applications like MixMonitor remain simpler and better documented. Once you do need that raw real-time audio path, the protocol itself is small enough to implement correctly in an afternoon — the production concerns covered above (timeouts, security boundaries, multi-tenancy, and scaling past a single process) are where most of the real engineering effort actually goes.