Agentic AI

The voice agent stack, layer by layer, and what to test in each

Amol Patil, Founder · 19 October 2026 · 12 min read

Explore with AI

ChatGPTPerplexityClaudeGoogleGrok

The short version

• Choose a voice agent stack one layer at a time, and keep each layer behind an interface you own. The best provider changes with the caller's language, the phone line and the latency budget.
• Phone calls arrive as narrowband 8 kHz audio. Test recognition and turn detection on recorded calls from your own lines, in every language you serve, before you trust a leaderboard.
• Use a cascaded pipeline for calls that look things up, call tools or capture long records. Use one speech to speech model where natural turn-taking matters most, and measure its speed instead of assuming it.
• Pin dated model versions, keep a fallback for every layer, time each stage at the median and the 95th percentile, and price whole calls.

We think a voice agent stack should be chosen one layer at a time, with each layer behind an interface you own, because the right provider changes with the caller's language, the phone network and the latency budget. A leaderboard can draw up the shortlist, but only a test on your own call audio can decide.

Five layers, and the test at each

A phone agent runs five layers in a loop: telephony, turn detection, speech to text, the language model and text to speech. The language model, or LLM, decides what to say or do, and a speech to speech model folds speech to text, the LLM and text to speech into one.

The cascaded loop, with the test at each stage

1. Live call: Phone, browser or WhatsApp. Leads to Telephony.
2. Recorded call: Replayed to compare providers. Leads to Telephony (replay).
3. Telephony: Test: audio format and sample rate. Leads to Turn detection, Speech to text.
4. Turn detection: Test: cut-offs and late replies. Leads to LLM and tools (end of turn).
5. Speech to text: Test: names, numbers, each language. Leads to LLM and tools (transcript).
6. LLM and tools: Test: right tool, right arguments. Leads to Text to speech (reply text).
7. Text to speech: Test: first audio, pronunciation. Leads to Telephony (reply audio).

An orchestration framework or a hosted platform runs the loop. Hosted platforms get a first call working fastest, and open-source frameworks give you the code for every stage. Either way, check that you can choose the provider at each layer and export call recordings, or you cannot test and switch later. Ours run on LiveKit Agents over SIP.

The main tools in each layer

Names change every few months, so treat this list as a shortlist to test, not a ranking.

The phone line sets the audio quality

Phone calls reach your agent as narrowband audio, sampled 8,000 times a second, or 8 kHz. Twilio's media streams carry it as mu-law, the G.711 audio format used on phone networks, and Telnyx uses the same by default. Telnyx and Vonage can stream 16 kHz audio, and Vonage recommends it for speech recognition.

Google's speech to text guidance says sampling rates under 16 kHz and lossy formats such as mu-law may reduce accuracy. It also tells you to send telephony audio at its native 8 kHz rather than resample it. So test recognition and turn detection on recorded calls from your own lines, not on studio speech.

Speech to speech models differ here too. GPT-Live accepts raw 8 kHz G.711 over a WebSocket connection, while Gemini 3.8 Live requires 16 kHz audio, so a bridge to Gemini must convert every call.

Browser and app calls sound better. WebRTC, the open standard for real-time audio in browsers, carries Opus, which Kwindla Hultman Kramer's voice AI primer rates well above G.711. Meta's WhatsApp Calling API now carries WhatsApp calls over WebRTC or SIP, though businesses with numbers in the United States, Canada, Egypt, Vietnam or Nigeria cannot start calls.

Turn detection decides when the agent speaks

Voice activity detection, or VAD, hears whether someone is speaking, not whether they have finished. Waiting for silence cuts people off when they pause to think, or leaves long gaps when they are done. Turn detection models also listen for meaning and intonation.

Check each detector's language list against your callers: the LiveKit turn detector covers 14 languages and Smart Turn v3.2 covers 23, both running alongside a VAD such as Silero. Then count cut-offs and slow replies on your recordings, language by language.

Keep the fast-moving layers swappable

Speech to text, the model and text to speech change fastest. Our own agents keep each behind a thin internal interface, so we can pick the best model for each language and a provider change touches one adapter.

As we argued in the model is a commodity, the model is the easiest part to replace. Route it through a gateway such as LiteLLM, which speaks one format to more than 100 providers and fails over to a backup. We route ours across OpenAI, Gemini, Claude and Grok. Keep a fallback for the other layers too.

Language coverage differs by vendor too. Rime's Coda speaks 10 languages and Cartesia's Sonic 3.6 speaks 44.

Pin exact versions, because vendors change what a model name points to. Cartesia's sonic-3.6 follows the latest stable snapshot, while sonic-3.6-2026-08-27 never changes. Soniox now sends stt-rt-v4 requests to its v5 model. Rime serves Mist v3 when a request omits the model ID or sends one it does not recognise. Plain Grok model names point to the latest stable release. Any of these can change how a number sounds mid-campaign.

Cascaded or speech to speech, per use case

OpenAI's voice agents guide describes three designs: a chained pipeline for control over each speech and text stage, a single speech to speech session, and GPT-Live, a full-duplex voice model that listens while it speaks and hands reasoning and tools to a separate backend.

Our view is to decide per use case. Calls that retrieve records, call tools or capture long forms, such as claims intake, suit a cascade. OpenAI's guide notes that a cascade can run policy checks and call internal systems before it generates any speech. Short calls where the feel of the conversation matters most, such as reception and booking, suit speech to speech.

Treat the speed gain as a claim to measure. The primer says OpenAI's and Google's speech to speech models are slower in practice than a well-tuned cascade, though far better on mixed-language speech, which matters for Indian callers.

Tool use varies as well. On the tau-voice customer service calls in the Artificial Analysis speech to speech leaderboard, Gemini 3.8 Live resolves 30 percent of tasks and its extended thinking version 69 percent. GPT-Live makes our case from the vendor side, letting the voice model talk while a text model you choose does the work.

If your callers are far from the US and Europe, time the extra round trip on real calls first.

Shortlist on leaderboards, decide on your own calls

Public leaderboards make a good shortlist. Artificial Analysis ranks speech to text by word error rate, the share of words transcribed wrongly. It ranks text to speech by blind listener votes, and speech to speech on reasoning, turn-taking and tool tasks. The Open ASR Leaderboard compares 86 systems on 12 datasets, as its paper explains.

Know what they leave out. The Artificial Analysis error rate covers about eight hours of audio: parliamentary speech, earnings calls and scripted prompts read aloud. Its methodology does not describe narrowband phone audio. The voice arena asks which sample sounds more natural, not whether your product names come out right. As we argued in benchmarks versus real work, the deciding test is built from your own cases.

In our transcription work, we benchmark Soniox, Speechmatics, AssemblyAI and Deepgram on the same recordings.

Where to start

Pull 50 to 100 recorded calls from your own lines, across languages, accents and background noise, and freeze them as a test set, as in keep your test set. Replay them through two candidates per layer, change one at a time and compare median and 95th percentile times, as OpenAI's guide suggests.

Time each stage, and the whole turn from the end of the caller's speech to the start of the reply, because the primer warns that vendor latency figures often count inference alone. Then price a whole call rather than a minute of audio, and add the simulated callers and native speakers from how to test a voice agent.

For how we approach this work, see our voice agents page and the FRIDAI case study. Designing the per-layer interfaces, fallbacks and tests is part of our agentic AI development work.

Sources

• OpenAI, Voice agents, Realtime API, Realtime transcription, Text to speech, Models, Pricing, Cost optimization, Your data and Telephony and SIP, accessed 19 October 2026, and the cookbook recipe Transcribing user audio with a separate Realtime request, November 2025
• Google Cloud, Speech-to-Text, Speech-to-Text best practices, Text-to-Speech, Gemini 3.8 Live, Gemini 3.8 Flash, Gemini Live API, Start and manage live sessions, Agent Platform pricing and Deployments and endpoints, accessed 19 October 2026
• Artificial Analysis, Speech to text leaderboard, Speech to text methodology, Text to speech leaderboard and Speech to speech leaderboard, accessed 19 October 2026, and AA-WER v2.0, 17 February 2026
• Vaibhav Srivastav and others, Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluation, arXiv, revised March 2026
• Hugging Face, Open ASR Leaderboard
• Kwindla Hultman Kramer and contributors, Voice AI and Voice Agents: An Illustrated Primer, 2026
• Twilio, Media Streams and Media Streams WebSocket messages
• Telnyx, Media streaming over WebSockets
• Vonage, WebSocket voice guide
• Meta, WhatsApp Business Calling API, updated 26 June 2026
• WebRTC project, WebRTC
• Exotel, home page
• Plivo, home page
• Tata Tele Business Services, Smartflo CPaaS
• Vobiz, home page
• LiveKit, Agents introduction and LiveKit turn detector
• Pipecat, pipecat and smart-turn on GitHub
• Vapi, home page
• Retell AI, home page
• Bland, home page
• Synthflow, home page
• Bolna, home page and bolna on GitHub
• Silero, silero-vad on GitHub
• Deepgram, Models and languages overview and Voices and languages
• AssemblyAI, Realtime speech to text API
• Speechmatics, Real-time speech to text
• Soniox, Speech to text models
• ElevenLabs, Speech to text and Models
• Sarvam, Speech to text API and Text to speech API
• Microsoft, Azure speech to text overview and text to speech overview
• Cartesia, Sonic 3.6
• Rime, Models
• Anthropic, Models overview
• SpaceXSI Docs, Grok models and pricing, updated 21 September 2026
• Groq, Supported models
• Cerebras, Model catalog
• BerriAI, LiteLLM on GitHub and Fallbacks