Evals and accuracy
How to test a voice agent with simulated callers and native speakers
- Test a voice agent with two programs. Simulated callers repeat every scenario on every change. Native speakers on real phones catch what simulation misses.
- A text eval of the transcript misses a slow reply, an agent talking over a caller, a misheard name and a voice that drifts mid-call.
- Count a scenario as passed only when every repeat passes, and check each field written to your CRM against the recording before the write.
- Word error rate counts a wrong digit the same as a dropped word, so also measure entity accuracy, latency to first word at the median and 95th percentile, and interruption handling.
We think a voice agent needs two test programs, not one. Simulated callers give volume and catch regressions on every change, native speakers on real phones catch the failures simulation misses, and a team with only one program releases the failures the other would have caught.
We build and run voice agents in production, so this plan follows the way they actually fail.
Why a transcript eval misses voice failures
A text eval checks what the agent said, not how long the caller waited or whether "fifteen" became "fifty". LiveKit's docs say its unit tests run in text mode and advise reserving audio runs "for turn-taking and speech-specific issues." Pipecat's eval docs say its audio mode catches problems with turn detection, sound-alike words and barge-in, meaning the caller talking over the agent.
Sierra's tau-voice benchmark gives voice agents the 278 customer service tasks that text agents face, then adds accents, background noise, dropped audio frames and interruptions.
Voice agents complete fewer tasks once the audio gets realistic
| Clean audio | Accents, noise and interruptions | |
|---|---|---|
| Google gemini-live-2.5-flash-native-audio | 31% | 26% |
| OpenAI gpt-realtime-1.5 | 49% | 35% |
| xAI grok-voice-agent | 51% | 38% |
Sierra reported in May that the best voice agent had since reached 67%, so read the chart as a pattern, not a ranking.
Two programs, two kinds of check
Functional checks ask whether the agent does the job. Non-functional checks ask how it sounds and holds up at scale. Each needs a manual program and an automated one.
What each program checks
| Native speakers on real phones | Simulated callers | |
|---|---|---|
| Does the job: scripted calls | Each main script, called by every tester before a release | Every scenario, repeated, on every change |
| Does the job: off-script calls | Rephrasing, objections, barge-in and topic changes | Personas that interrupt, ramble, refuse or change their mind |
| Does the job: captured fields | Each field checked against the recording | Tool calls, arguments and CRM writes checked against expected values |
| Sound: voices and accents | Genders, ages and accents that match your callers | Synthetic accents, as varied as the speech engine allows (Partly) |
| Sound: phones and rooms | Real handsets, carriers, speakerphones and noisy rooms | 8 kHz phone audio with added noise and packet loss |
| Sound: tone and drift | Naturalness, tone, pronunciation, and a voice that shifts accent or gender | Pronunciation and pace scores, while tone still needs a person (Partly) |
| Scale: speed | Silences that feel long, noted per call | Latency to first word at the median and 95th percentile, and every silence over budget |
| Scale: load, cost and limits | Too few calls to measure (No) | Peak concurrency, cost per call and session limits |
Native speakers on real phones
- Does the job: scripted calls
- Each main script, called by every tester before a release
- Does the job: off-script calls
- Rephrasing, objections, barge-in and topic changes
- Does the job: captured fields
- Each field checked against the recording
- Sound: voices and accents
- Genders, ages and accents that match your callers
- Sound: phones and rooms
- Real handsets, carriers, speakerphones and noisy rooms
- Sound: tone and drift
- Naturalness, tone, pronunciation, and a voice that shifts accent or gender
- Scale: speed
- Silences that feel long, noted per call
- Scale: load, cost and limits
- Too few calls to measure (No)
Simulated callers
- Does the job: scripted calls
- Every scenario, repeated, on every change
- Does the job: off-script calls
- Personas that interrupt, ramble, refuse or change their mind
- Does the job: captured fields
- Tool calls, arguments and CRM writes checked against expected values
- Sound: voices and accents
- Synthetic accents, as varied as the speech engine allows (Partly)
- Sound: phones and rooms
- 8 kHz phone audio with added noise and packet loss
- Sound: tone and drift
- Pronunciation and pace scores, while tone still needs a person (Partly)
- Scale: speed
- Latency to first word at the median and 95th percentile, and every silence over budget
- Scale: load, cost and limits
- Peak concurrency, cost per call and session limits
The first three rows are functional checks, the other five non-functional.
What native speakers on real phones catch
Speech recognition can work worse for some callers. In a 2020 study of five commercial systems, the average word error rate, the wrong, missing and extra words as a share of words spoken, was 0.35 for Black speakers against 0.19 for white speakers, and 0.41 for Black men against 0.30 for Black women. Recruit testers whose voices match your callers, calling from cars, kitchens and streets.
Voices can drift too. OpenAI's GPT-4o system card reports rare test cases where the model imitated the user's voice, and OpenAI added a classifier that blocks speech in any voice but the chosen preset. ElevenLabs says its stability setting controls "the randomness between each generation". Treat an accent or perceived gender that changes mid-call as a release blocker, because no transcript shows it.
What simulated callers catch
The same scenario can pass and then fail, because an agent's model does not always give the same output for the same input, as LiveKit notes. The tau-bench paper measures this with pass^k, the chance of succeeding on all k tries. On the paper's retail tasks, agents such as gpt-4o scored below 25% on pass^8.
So run each persona several times and count a scenario as passed only when every repeat passes. Run the suite on every prompt, model, voice or provider change, and treat each change as a release. Check the tool calls and the record the agent wrote as well as its words.
Measure latency to first word, from the caller's last word to the agent's first sound, at p50 and p95, the median and the 95th percentile. Across ten languages, the mean gap between conversational turns was 208 milliseconds. One call in 20 is slower than the p95, which the median never shows.
Twilio's media streams always deliver 8,000 Hz mu-law audio on one channel, a compressed phone format, so run simulated callers through the same format with noise and dropped packets, as tau-voice does.
Limits need tests too. Retell gives pay-as-you-go workspaces 20 concurrent calls by default, rejects outbound calls over the limit unless burst is enabled, and ends calls after an hour by default. Google limits Gemini Live audio-only sessions to about 15 minutes unless you compress the session context. Load-test at your peak, push one call past each limit, and log cost per call.
Five failures to test for by name
We have hit the first, third and fourth on voice agents we run, along with a voice that drifted mid-call, perceived gender included.
- Voice activity detection, which decides when someone is speaking, so sensitive that it cuts callers off at a short pause or treats background noise as speech. LiveKit's tuning guide lists "Agent cuts users off mid-thought" as a symptom. Test with callers who pause to read out a card number.
- Endpointing, the decision that the caller has finished, set to wait too long, so every reply follows a long pause. Compare latency to first word after short answers and after long ones.
- The same reply after a caller rephrases. Have the caller reword a request after a non-answer, and fail the run if the agent repeats itself word for word.
- CRM fields with no evidence in the recording. In a 2024 study, about 1% of Whisper transcriptions held whole phrases absent from the audio, more often for speakers with long pauses, and a tau-voice agent said "I've updated your shipping address" without calling any tool. Check each field against the recording before the write and refuse any field it does not support, as our own agents do after every call.
- Misheard numbers, dates, names and email addresses. In tau-voice, agents failed to transcribe names and emails even when callers spelled them letter by letter. Have the agent read each one back before acting, and test with callers who correct it.
Measure more than word error rate
Word error rate is the standard speech recognition metric, but the Conformer-1 paper notes that it "penalizes all types of errors equally" and misses names and places, so a wrong account digit costs the same as a dropped "the".
Measure it on your own calls, then add entity accuracy, the share of numbers, dates, names and emails captured exactly, and interruption handling: how fast the agent stops when talked over, whether its next reply answers the caller, and how often an "uh-huh" stops it.
Tools that run simulated callers
Five dedicated voice testing platforms, in their own words
| Hamming | Coval | Cekura | Bluejay | Roark | |
|---|---|---|---|---|---|
| How it builds test calls | Scenarios generated from your agent's system prompt; claims 50K+ concurrent test calls | Thousands of realistic conversations before launch, with noisy audio | Thousands of synthetic conversations before go-live | Digital Humans across voice, chat and text, and replays of production calls | Callers built from your real call types, such as the interrupter |
| Conditions and checks it names | Accents, interruptions, background noise, keypad tones, phone menus and time to first word | Frustrated callers, interruptions, slow talkers and callers who ignore the script | Red teaming for jailbreaks, personal data leaks and off-script behavior; turn latency | Edge cases, regressions and security; scores for tone, accuracy and conversion | Adversarial callers; claims 45 languages and accents; pronunciation and pace |
| Releases and live calls | Blocks failing builds from shipping; replays a golden set of calls to catch drift | A quality gate per release; production evals, with high-stakes calls routed to human reviewers | Gates every deploy on the suite; drift detection on live calls | Load tests at scale; A/B tests of prompts and flows | Each change compared with the last green run; every live call scored |
Hamming
- How it builds test calls
- Scenarios generated from your agent's system prompt; claims 50K+ concurrent test calls
- Conditions and checks it names
- Accents, interruptions, background noise, keypad tones, phone menus and time to first word
- Releases and live calls
- Blocks failing builds from shipping; replays a golden set of calls to catch drift
Coval
- How it builds test calls
- Thousands of realistic conversations before launch, with noisy audio
- Conditions and checks it names
- Frustrated callers, interruptions, slow talkers and callers who ignore the script
- Releases and live calls
- A quality gate per release; production evals, with high-stakes calls routed to human reviewers
Cekura
- How it builds test calls
- Thousands of synthetic conversations before go-live
- Conditions and checks it names
- Red teaming for jailbreaks, personal data leaks and off-script behavior; turn latency
- Releases and live calls
- Gates every deploy on the suite; drift detection on live calls
Bluejay
- How it builds test calls
- Digital Humans across voice, chat and text, and replays of production calls
- Conditions and checks it names
- Edge cases, regressions and security; scores for tone, accuracy and conversion
- Releases and live calls
- Load tests at scale; A/B tests of prompts and flows
Roark
- How it builds test calls
- Callers built from your real call types, such as the interrupter
- Conditions and checks it names
- Adversarial callers; claims 45 languages and accents; pronunciation and pace
- Releases and live calls
- Each change compared with the last green run; every live call scored
Each vendor's own description, from its home page on 19 October 2026: Hamming, Coval, Cekura, Bluejay and Roark.
Voice platforms and frameworks ship their own tests, and most offer a cheaper text mode beside audio. Vapi simulations run in a text chat mode or a voice mode, for as many iterations as you choose. Retell pairs text simulation tests with web calls for real audio and interruptions, and phone calls for carrier audio, keypad tones and transfers.
LiveKit's Agents framework is open source, and its Agent Simulations, generally available on LiveKit Cloud since October, can run over audio. Pipecat, also open source, includes the Evals above. Outside the frameworks, LangWatch's open-source Scenario has voice adapters for ElevenLabs, OpenAI Realtime, Twilio, Pipecat and Gemini Live. Whichever you pick, keep the scenarios in your own repository, so you can switch tools.
Outbound calls in the US
On 8 February 2024, the FCC ruled that calls made with AI-generated voices are "artificial" under the Telephone Consumer Protection Act, so they face the same rules, including prior express written consent before telemarketing robocalls. Test that an outbound agent calls only numbers with recorded consent. This is not legal advice. Indian rules are in our India post.
The test plan, in order
Test a voice agent in this order
BothWrite scenarios from real calls
Use recorded calls, or role-played ones before launch. Give each scenario a goal, the fields to capture and the action that ends it.
ManualRun a native-speaker round
Testers who match your callers call on their own phones and carriers, from quiet and noisy rooms, on and off script.
ManualCheck every captured field against the audio
Each unsupported field or misheard number becomes a new scenario.
AutomatedTurn testers into personas
Encode their habits, such as interrupting, rephrasing, refusing or spelling a name.
AutomatedRun every persona several times
A scenario passes only when every repeat passes, in text and in 8 kHz audio.
AutomatedGate every change
Run text tests on every pull request and audio tests before each release, for any prompt, model, voice or provider change.
AutomatedTest load, limits and cost
Call at your expected peak, with a call generator such as SIPp if your phone lines use SIP. Push one call past each session limit, and log cost per call.
ManualRepeat the human round on big changes
A new voice, model or speech provider gets a fresh round before callers hear it.
BothFeed live calls back
Every failed production call becomes a scenario.
Where to start
Take ten recent calls, write each as a scenario, and run each one five times. The share that passes all five times is your baseline, and the failures are your first fixes. For the layers behind them, see the voice agent stack. extendfuture's approach, with the FRIDAI voice assistant case study, is on our voice agents page.
Sources
- Soham Ray, Keshav Dhandhania, Victor Barres and Karthik Narasimhan, τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains, arXiv, March 2026
- Sierra, τ-voice: benchmarking real-time voice agents, 1 May 2026
- Shunyu Yao, Noah Shinn, Pedram Razavi and Karthik Narasimhan, τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, arXiv, June 2024
- LiveKit, Debugging and testing overview, Turn-taking tuning, Testing voice agents with Agent Simulations (8 October 2026) and livekit/agents
- Pipecat, Pipecat Evals and pipecat-ai/pipecat on GitHub
- Vapi, Simulations overview, docs
- Retell AI, Testing overview and concurrency limits, docs
- Google, Limits and specifications of the Live API, Firebase AI Logic docs, updated 13 October 2026
- Twilio, Media Streams WebSocket messages, docs
- Allison Koenecke and colleagues, Racial disparities in automated speech recognition, PNAS, March 2020
- Allison Koenecke and colleagues, Careless Whisper: Speech-to-Text Hallucination Harms, ACM FAccT, 2024
- Kevin Zhang and colleagues, Conformer-1: Robust ASR via Large-Scale Semisupervised Bootstrapping, arXiv, April 2024
- OpenAI, GPT-4o System Card, arXiv, October 2024
- ElevenLabs, Get default voice settings, API reference
- Tanya Stivers and colleagues, Universals and cultural variation in turn-taking in conversation, PNAS, June 2009
- Federal Communications Commission, FCC Makes AI-Generated Voices in Robocalls Illegal, 8 February 2024
- Vendor home pages, read 19 October 2026: Hamming, Coval, Cekura, Bluejay and Roark
- LangWatch, scenario on GitHub
- SIPp, SIPp/sipp on GitHub