Building a GPT-4 Powered IVRon Asterisk with AGI & Whisper ASR
End-to-end implementation: Asterisk AGI calls Whisper for transcription, passes intent to GPT-4, synthesises a response via TTS, and routes the call — all in under 3 seconds per turn.
1–3s
end-to-end latency
$0.05
per 3-min call
6 turns
max conversation
95%
Whisper accuracy
By Kaushik Parmar · Founder & VoIP Architect, CelloIP Technologies · February 22, 2026 · 20 min read
How do you build a GPT-4 IVR on Asterisk?
Build a GPT-4 IVR by running a Python FastAGI server that: records caller speech with Asterisk RECORD FILE, sends audio to OpenAI Whisper for transcription, passes the transcript to GPT-4 for intent extraction and response, synthesises the response with TTS, and plays it back — looping for multi-turn conversations.
Table of Contents
AI IVR vs Traditional IVR
Traditional IVR (Menu-Driven)
AI IVR (Intent-Driven)
AI IVR Pipeline Architecture
Inbound Call
Asterisk INVITE
Speech Capture
RECORD FILE (8s)
Whisper ASR
Transcription 300–800ms
GPT-4o-mini
Intent + response 400–1200ms
ElevenLabs TTS
Synthesis 200–500ms
Play + Route
STREAM FILE or Dial
| Component | Technology | Latency Added |
|---|---|---|
| Asterisk AGI | Python FastAGI (socket) | ~0ms (local) |
| Speech capture | Asterisk RECORD FILE | User-dependent |
| ASR | OpenAI Whisper API (turbo) | 300–800ms |
| LLM | GPT-4o-mini | 400–1200ms |
| TTS | ElevenLabs / AWS Polly | 200–500ms |
| Total per turn | Full pipeline | ~1–3 seconds |
FastAGI Server: Core Conversation Loop
The FastAGI server runs on port 4573 and handles the full conversational turn — record, transcribe, LLM, TTS, play. Asterisk connects over TCP and issues commands via the AGI protocol.
import asyncio, openai, boto3
from asterisk.agi import AGI
async def handle_call(agi: AGI):
agi.answer()
history = [{"role": "system", "content":
"You are a helpful assistant for CelloIP Technologies. "
"Answer questions about our VoIP services. "
"If the caller wants to speak to a human, say TRANSFER_SALES."}]
for turn in range(6): # max 6 turns
# Record caller speech (max 8 seconds, silence threshold 3s)
agi.record_file('/tmp/caller_input', 'wav', '#', 8000, 0, 3)
# Transcribe with Whisper
with open('/tmp/caller_input.wav', 'rb') as f:
transcript = openai.audio.transcriptions.create(
model="whisper-1", file=f).text
if not transcript.strip():
agi.stream_file('sorry-could-not-hear-you')
continue
# LLM intent + response
history.append({"role": "user", "content": transcript})
resp = openai.chat.completions.create(
model="gpt-4o-mini", messages=history)
reply = resp.choices[0].message.content
history.append({"role": "assistant", "content": reply})
if "TRANSFER_SALES" in reply:
agi.exec('Dial', 'SIP/sales_queue')
return
# TTS → WAV → play
tts_audio = synthesise_tts(reply) # calls Polly or ElevenLabs
agi.stream_file(tts_audio)
agi.stream_file('transferring-to-agent')
agi.exec('Dial', 'SIP/support_queue')Asterisk Dialplan Integration
Connecting the AGI server to inbound calls is a single-line dialplan entry. Asterisk connects to your FastAGI server over TCP and streams AGI commands.
[from-trunk] exten => _.,1,NoOp(AI IVR inbound) same => n,Answer() same => n,AGI(agi://127.0.0.1:4573/ai_ivr) same => n,Hangup() ; Fallback if AGI server is down exten => _.,n,Dial(SIP/support_queue,30) exten => _.,n,VoiceMail(s@default) exten => _.,n,Hangup()
Reducing Latency Below 1.5 Seconds
ASR Optimisation
Whisper large-v3: 1.2s
✅ Whisper turbo: 350ms
850ms savedLLM Optimisation
GPT-4o: 2.0s / 3x cost
✅ GPT-4o-mini: 600ms
1.4s + 66% cheaperTTS Streaming
Generate all → play
✅ First chunk plays while rest generates
500ms perceived gainOn-Premise: Llama 3 + Whisper.cpp + Coqui TTS
For regulated industries (healthcare, finance, legal) where call audio cannot leave your servers:
| Cloud Component | On-Premise Alternative | Cost Saving | Latency Impact |
|---|---|---|---|
| OpenAI Whisper API | Whisper.cpp (local) | ~$0.006/min saved | +200ms |
| GPT-4o-mini | Llama 3 via Ollama | ~$0.002/turn saved | +300ms |
| ElevenLabs TTS | Coqui TTS | ~$0.01/min saved | +100ms |
# Whisper.cpp (local ASR — GPU or CPU) git clone https://github.com/ggerganov/whisper.cpp cd whisper.cpp && make ./models/download-ggml-model.sh base.en # or large-v3-turbo # Ollama for Llama 3 local LLM curl -fsSL https://ollama.ai/install.sh | sh ollama pull llama3:8b # 8B model — fast enough for IVR # Coqui TTS pip install TTS tts --model_name tts_models/en/ljspeech/tacotron2-DDC
Cost Breakdown per Month
1,000 calls/mo
Small business IVR
Cloud APIs
$50
On-premise
$0 (+ GPU server)
10,000 calls/mo
Mid-market call centre
Cloud APIs
$500
On-premise
$150 (GPU cloud)
100,000 calls/mo
Enterprise / carrier
Cloud APIs
$5,000
On-premise
$800 (own GPU)
Frequently Asked Questions
QHow much does AI IVR cost per minute?
At current pricing: Whisper ~$0.006/min, GPT-4o-mini ~$0.002/turn, ElevenLabs ~$0.01/min. A 3-minute call with 4 turns costs approximately $0.05. For 10,000 calls/month that is $500 in API costs.
QCan this run on-premise without OpenAI?
Yes. Replace Whisper API with a local Whisper.cpp instance, replace GPT-4 with Llama 3 (via Ollama), and replace ElevenLabs with Coqui TTS. Latency increases but there are zero API costs and no data leaves your server.
QHow do you handle caller silence or background noise?
Use Asterisk's RECORD FILE with a silence threshold parameter (3 seconds stops recording). For noisy environments, preprocess audio with noise reduction before sending to Whisper.
QCan the AI transfer calls to a human?
Yes. Include a transfer intent in the system prompt — when GPT-4 returns a TRANSFER keyword, the AGI script executes an Asterisk Dial command to the appropriate queue or extension.
Related Articles
Official References
Build Your AI IVR with CelloIP
CelloIP Technologies builds production AI IVR systems on Asterisk and FreeSWITCH — from GPT-4 cloud pipelines to fully on-premise Llama 3 deployments for regulated industries.