Evals and accuracy
India is the hardest test for a voice agent
- India is the hardest test for a voice agent. Callers switch languages mid-sentence, say numbers the Indian way and call over narrowband lines from noisy places, and outbound calls face strict rules.
- English-first leaderboards will not tell you whether you pass. Score each model on real phone audio from your own callers, in their languages, against one transcript script.
- Indian makers fill gaps the global stack leaves: Sarvam's speech, language and voice agent models, AI4Bharat's open speech data and benchmarks, and the government's Bhashini platform.
- A September 2026 amendment from India's telecom regulator makes senders declare calls placed by software, including artificial voices, to their operator in advance. The readiness check below shows what to fix first.
We think India is the hardest market for a voice agent, and the most useful one to test against. An agent that understands a caller mixing Hindi and English on a noisy phone line, reads a loan amount back in lakh and follows India's calling rules will cope almost anywhere, while a stack chosen from English leaderboards cannot tell you whether it passes.
We build and run voice agents in Hindi, Hinglish, Indian English and Telugu, with callers switching language mid-call.
This post covers what is specific to India. The general test program is in how to test a voice agent, and the global stack, layer by layer, in the voice agent stack.
Why English-first scores mislead
India's constitution lists 22 scheduled languages, and the 2011 census counted 99 more, according to a government backgrounder. Callers also code-mix, switching languages inside one sentence, as in Hinglish, which mixes Hindi and English.
A leaderboard score describes its own test set. Voice of India, a benchmark from IIT Madras and Josh Talks, is closer to a real call. It holds 536 hours of unscripted phone conversations from 36,691 speakers in 15 Indian languages.
This ranking covers one language on one benchmark, and your callers, lines and script can reorder it. We would choose a model with test calls from your own callers, as we argued in benchmarks versus real work. Global providers sit close behind the leader on Hindi, so test them alongside the Indian ones.
Script, numbers and read-back
Code-mixing also distorts the scores. Sarvam's guide to evaluating Indian speech recognition shows word error rate, the share of words transcribed wrong, marking the English word doctor written in Devanagari as a mistake against the same word in Latin letters. It also treats 500 written in digits, in Devanagari digits and in Hindi words as three different answers.
Sarvam's Saaras model can return native script, romanised Latin script, a code-mixed form that keeps English words in Latin letters, or an English translation. Pick one form for transcripts, write it into your test set, and score every candidate model against it, Indian or global.
Numbers come out the Indian way. A lakh is 100,000, a crore is 10 million and dhai means two and a half. Saade teen hazaar means 3,500, and a study of Hindi numerals notes that such and-a-half forms tend to be heard as approximate. A transcript can be word-perfect while the amount in the loan form is wrong, which is why our own agents check every captured field against the call recording before it is saved.
Every language's test set also needs names, addresses with PIN codes, India's postal codes, email addresses spelled letter by letter, and phone and account numbers. Have the agent read each one back and wait for a yes before it acts. Sarvam's guide proposes an entity preservation score, the share of names, numbers and dates kept intact, with a bar of 0.95 for banking, medical and navigation uses.
The Indian makers to test
Three Indian sources are worth testing. Sarvam AI is one of 12 organisations the government selected under the IndiaAI Mission to build indigenous foundation models, with 246.72 crore rupees in financial and compute support, according to the government's Press Information Bureau. AI4Bharat is a research lab at IIT Madras that releases much of its work as open source. Bhashini is the government's language platform.
Indian building blocks for a voice agent
| Sarvam AI | AI4Bharat and Bodhan AI | Bhashini | |
|---|---|---|---|
| Speech to text | Saaras V4, launched 25 September 2026, for 22 Indian languages and English, tuned for 8 kHz phone audio, per Sarvam | IndicConformer, 22 languages, whose card now recommends Indic-Transcribe, released by Bodhan AI and AI4Bharat in September 2026 | Voice services in 22 languages and text services in 36 |
| Text to speech | Bulbul V4 Flash, 13 October 2026, in 11 languages, with under 250 ms to first audio, per Sarvam | AI4BTTS voices for Indian languages, and the Rasa dataset for training voices | Part of its language services |
| Open releases | Sarvam 30B and 105B language models under Apache 2.0, trained in India on IndiaAI Mission compute | IndicVoices, 12,000 hours of speech from 22,563 speakers in 208 districts | Hosts more than 350 AI models and datasets |
| How you run it | API, or managed Voice Agents, whose data Sarvam says stays in India | Open weights on Hugging Face to host yourself | Government platform, launched in 2022 under the National Language Translation Mission |
Sarvam AI
- Speech to text
- Saaras V4, launched 25 September 2026, for 22 Indian languages and English, tuned for 8 kHz phone audio, per Sarvam
- Text to speech
- Bulbul V4 Flash, 13 October 2026, in 11 languages, with under 250 ms to first audio, per Sarvam
- Open releases
- Sarvam 30B and 105B language models under Apache 2.0, trained in India on IndiaAI Mission compute
- How you run it
- API, or managed Voice Agents, whose data Sarvam says stays in India
AI4Bharat and Bodhan AI
- Speech to text
- IndicConformer, 22 languages, whose card now recommends Indic-Transcribe, released by Bodhan AI and AI4Bharat in September 2026
- Text to speech
- AI4BTTS voices for Indian languages, and the Rasa dataset for training voices
- Open releases
- IndicVoices, 12,000 hours of speech from 22,563 speakers in 208 districts
- How you run it
- Open weights on Hugging Face to host yourself
Bhashini
- Speech to text
- Voice services in 22 languages and text services in 36
- Text to speech
- Part of its language services
- Open releases
- Hosts more than 350 AI models and datasets
- How you run it
- Government platform, launched in 2022 under the National Language Translation Mission
Figures from each maker's own pages. Bhashini figures from the government backgrounder linked above.
For English spoken with Indian accents, AI4Bharat's Svarah benchmark has 9.6 hours from 117 speakers across 65 districts, and Vistaar collects 59 benchmarks across Indian languages and domains. Use them as a first filter before recording your own calls.
Narrowband lines on many networks
Phone audio is narrowband. Exotel streams call audio at 8 kHz, or 8,000 samples a second, unless you ask for more, according to its documentation. Plivo's streaming guide calls 8 kHz the native telephony format and keeps 16 kHz for wideband recognition. AI4Bharat's IndicConformer expects 16 kHz and resamples anything else. Score every model on recordings of real calls, not studio audio.
Several telephony providers that serve India connect calls to an agent you build. Exotel, Plivo, Vobiz and Tata Tele Business Services' Smartflo stream call audio to it, and Ozonetel sells its own voice AI agents for contact centres. Plivo notes that Indian numbers need know-your-customer checks, so start that paperwork early.
Network scale and audio quality
Network
1,293 million
wireless mobile subscribers in India at the end of August 2026
Network
42.1%
of those subscribers in rural areas
Measured
15.3% to 25.2%
ElevenLabs Scribe's error rate in Voice of India, from the best to the worst quarter of recordings by audio quality
Vendor claim
Under 300 ms
voice latency that Exotel claims for real-time calls
Place test calls on each major network, Jio, Airtel, Vodafone Idea and BSNL, from low-end Android phones as well as new ones, indoors and on the street.
Host the speech and model layers in an Indian region, and check that the model you need is offered there before you design around it. Google's Vertex AI location list shows Gemini 2.5 Flash in Mumbai but its Gemini Live audio models in no Asia Pacific region, and warns that its global endpoint gives no control over where a request is processed.
The rules: TRAI, DPDP and RBI
Outbound calls in India must follow rules on number series, declarations and consent that a stack built for another market will not handle. Most are set by TRAI, India's telecom regulator.
The calling rules a voice agent must meet
February 2025Commercial calls come from designated number series
TRAI's 2025 amendments bar ordinary 10-digit numbers for telemarketing. Promotional calls use the 140 series and service and transactional calls the 1600 series.
July 2026The series depends on who calls and why
TRAI mandates 1600 numbers for service and transactional calls to existing customers from banks and other firms regulated by RBI, SEBI, IRDAI and PFRDA, India's financial regulators, and from government. Customers can block 140 promotional calls by sector on the Do Not Disturb registry.
September 2026Automated calls must be declared first
The Third Amendment counts calls placed by software, including artificial voices, as application-to-person calls. From 60 days after Gazette publication, senders must declare them and their caller numbers to their operator in advance, or the calls are treated as unsolicited. Its text places service and transactional calls from other sectors on the 1601 series.
January 2027Collection calls keep to hours and get recorded
RBI, the central bank, has barred recovery agents from calling borrowers before 8 a.m. or after 7 p.m. since 2022. For commercial banks, amended directions from 1 January 2027 add recorded calls, kept for six months, with the borrower told that the call is recorded.
May 2027Data notices and retention apply
The Digital Personal Data Protection Act was enacted on 11 August 2023. Under its Rules, published in November 2025, the rules on notices, security and retention apply 18 months after publication. A notice must itemise the personal data and its purpose so the person can give informed consent, and processing logs must be kept for at least a year.
We would add one practice that the rules above do not spell out. Say in the first sentence that the caller is talking to an AI, and how to reach a person. Where a call feeds a decision about a person, such as a loan, design the human review as a control, as we set out in human review is a legal control.
This is not legal advice. Have counsel check your call flows before launch, because TRAI amended its rules in February 2025 and again in September 2026.
Test with the people who will call
Build a native speaker panel for each language, spread across regions, genders and age groups. In Voice of India, average error by district ranged from about 4% in Nainital to 44% in Mannarakkat, and models did slightly better on women's speech than on men's. A panel from one city will overstate the agent's accuracy.
Write test cases from real phrasing, not translated English scripts. IndicVoices is 76% extempore speech, people talking freely rather than reading, and your test set should sound the same.
The readiness check
The 13 questions come in four groups, and the first group with a no gives your result and what to fix first. Language comes first, because an agent that mishears a caller fails every later step. The phone network comes next, because a model that passes on clean audio can fail on a real line. Rules come before launch, and operations keep the agent passing after it.
Is your voice agent ready for Indian callers?
Answer for one agent and one call type. Count a yes only if it is true today.
Language
1Do you have a test set of real caller phrasing in each language you serve, including code-mixed speech?
2Have you chosen one transcript script, native or romanised, and do you score every model against it?
3Does the test set include amounts in lakh and crore, half numbers such as dhai, digit strings, names, addresses and email addresses?
4Does the agent read back every number, name and address and wait for a yes before it acts?
Phone network
5Have you scored the agent on recordings of real 8 kHz calls, not studio audio?
6Have you placed test calls on each major network, from low-end phones as well as new ones, in noisy places?
7Are the speech and model layers hosted in an Indian region, with latency measured from Indian networks?
Rules
8Do outbound calls use the right series, 140 for promotional calls and 1600 or 1601 for service and transactional calls?
9Have you declared your automated calls and caller numbers to your telecom operator?
10Does every call disclose the AI at the start, give notice for the personal data it collects, and keep records for the required period?
Operations
11Do native speakers from each region, gender and age group review a sample of live calls every week?
12Does every failed call become a new case in the test set?
13Can a caller reach a person at any point, with the call's context passed on?
13 questions. Answer yes only for what is true today.
Where to start
Pick the language most of your callers use, record a few dozen real calls in it, and score two or three models, Indian and global, against one transcript script and an entity check with read-back. extendfuture's approach to voice agents, with human review where accuracy matters, is on our voice agents page.
Sources
- Department of Official Language, Ministry of Home Affairs, Languages included in the Eighth Schedule of the Indian Constitution
- Press Information Bureau, AI for culture and languages, 9 February 2026; Sarvam AI backgrounder, 21 February 2026; DPDP Rules, 2025 notified, 17 November 2025
- Bhogale and others, IIT Madras and Josh Talks, Voice of India, arXiv, April 2026, revised July 2026
- Sarvam AI, Evaluating Indian Language ASR, 2 April 2026; Introducing Saaras V4, 25 September 2026; Bulbul V4 Flash, 13 October 2026; Sarvam 30B and 105B, 6 March 2026
- Sarvam AI docs, Saaras, Voice Agents and Smartflo telephony, accessed October 2026
- AI4Bharat, homepage, IndicVoices, Svarah and IndicConformer
- Bhogale and others, Vistaar, Interspeech 2023
- Bodhan AI and AI4Bharat, Indic-Transcribe core, September 2026
- John T. Platts, A Dictionary of Urdu, Classical Hindi, and English, entries lakh, karor and dhai, Digital South Asia Library
- Shaunak Phadnis, 'and-a-half' numeral constructions in Hindi, FASAL 12, April 2023
- Exotel, Working with the Stream and Voicebot Applet, September 2025; AgentStream; homepage
- Plivo, Audio streaming overview
- Vobiz, telephony for AI voice agents
- Ozonetel, Voice AI Agents
- Google Cloud, Vertex AI generative AI locations
- TRAI, press release 11/2025, 12 February 2025; 91/2026, 10 July 2026; 119/2026, 18 September 2026
- TRAI, TCCCPR Third Amendment Regulations, 2026, 18 September 2026; subscription data for August 2026, 28 September 2026
- Ministry of Electronics and Information Technology, Digital Personal Data Protection Rules, 2025, November 2025
- Reserve Bank of India, recovery agents circular, 12 August 2022; Responsible Business Conduct Fourth Amendment Directions, 6 August 2026