Multilingual voice AI: handling accents and code-switching

The hard problem in multilingual voice AI is not translating between languages — it is handling a single caller who opens in accented English, drops into Hindi halfway through a sentence, says a number in one language and a currency in another, and expects the agent to follow all of it in real time. Most platforms can demo "we support 30 languages." Far fewer survive a real call where the languages are mixed, the accent is regional, and the caller never signals a switch. This post is about that gap.

For the market-specific view, pair this with multilingual voice agents for India; for the bigger picture of how agents work, start from the complete guide to AI voice agents.

Why accents and code-switching are hard

Speech recognition is trained on distributions of audio. It is confident where it has heard a lot and shaky at the edges — and accents and code-switching live at the edges.

Accents move the acoustic signal away from whatever the transcription model saw most. A model trained heavily on US English will mishear an Indian, Nigerian, or Scottish speaker not because those accents are "wrong" but because they are under-represented in its training. The failure is quiet: the agent doesn't crash, it just mis-transcribes a word and answers the wrong question.

Code-switching — mixing languages inside one utterance — breaks a deeper assumption. Many pipelines pick one language per turn and transcribe the whole thing in it. But "मुझे EMI का due date change karna hai" is not a Hindi sentence or an English sentence; it is both, in the natural register millions of people actually speak. Force it into a single language and you get garbled output in either direction.

There is a subtler failure mode we have seen in production: the wrong-script transcription. The recognizer hears the words correctly but writes them in the wrong script — Hindi words rendered in Latin letters, or English words pushed into Devanagari. The meaning is technically there, but downstream text matching, tool calls, and the language model's own reasoning degrade because the text isn't in the form anything expects. Robust multilingual systems normalize script explicitly rather than trusting the recognizer's default.

Native language packs vs accented English

A common shortcut is to treat everything as English and lean on the model's tolerance for accents. It is cheaper and simpler, and for lightly-accented speakers it works. But there is a real quality ceiling: accented-English handling degrades exactly where the accent is strongest and where callers are most likely to code-switch into their first language.

The alternative is native language support — transcription and synthesis models built for the specific language, not English models stretched to cover it. For Indian languages, specialist engines tuned for Hindi, Hinglish, Indian English, and regional languages materially outperform a general English model on the same audio. The practical rule:

• Accented English is acceptable when your callers mostly speak one language with an accent, and switches are rare.
• Native language packs are necessary when callers genuinely speak in their own language, or mix freely, and you care about the tail of the accent distribution — which, in most real markets, is a large share of your callers.

The deeper point is that this should be a choice you can make per deployment, not a limitation baked into the platform. The models and languages available under the hood decide whether you even have the option.

Pronunciation and normalization: numbers, currency, dates

Multilingual quality is won and lost on the small, unglamorous stuff — and numbers are the worst offender.

A caller says a phone number, an amount, or a date, and the agent has to both understand it and say it back correctly. This is harder than it looks:

• Numbers spoken in mixed languages. "Twenty-five thousand" might arrive as a mix of English and a local language, or in a local numbering system (lakhs and crores rather than thousands and millions). Naive normalization mangles it.
• Currency. "₹1,499" should be voiced as "one thousand four hundred ninety-nine rupees," not "rupee one four nine nine," and the right phrasing differs by language.
• Digit strings vs quantities. A phone number is read digit by digit; an amount is read as a quantity. The system has to know which is which from context.
• Dates and times. Formats and spoken conventions vary by locale.

When a message must be legally or commercially exact — a payment amount, a disclosure, an offer — this is another reason to prefer a pipeline with classic text-to-speech, where what you wrote is what plays, over a speech-to-speech model that paraphrases. We cover that tradeoff in the complete guide.

Language detection: when and how to switch

If the agent supports multiple languages, something has to decide which one the caller is speaking — and when to switch. The naive version switches the moment it thinks it heard a different language, which produces flapping: the agent lurches between languages on every ambiguous word, which is far worse than politely staying in one.

The disciplined version treats a switch as a decision that needs evidence. A useful design principle: require a couple of consecutive turns in the new language before committing to the switch. One stray English word inside a Hindi sentence is code-switching, not a language change; a caller who has now said two full turns in English is genuinely switching. Building in that hysteresis is the difference between an agent that feels stable and one that feels like it's arguing with itself.

Which brings up an architectural constraint worth knowing before you buy.

When speech-to-speech models limit language coverage

Speech-to-speech (realtime) models are wonderfully natural, but they handle language internally — you feed them audio and they respond, and you don't get to route the transcription to a language-specialist engine or force a script. That has two consequences for multilingual work:

1. You inherit the model's languages. If the frozen model doesn't cover a language or handles a regional accent poorly, you cannot swap in a better transcriber for that call the way you can in a pipeline. Your coverage is the model's coverage.
2. Explicit language switching is limited. In a pipeline you can run automatic detection and switching with tuned rules; with a single speech-to-speech model, language handling is whatever the model does natively, and you lose the knobs.

None of this makes speech-to-speech the wrong choice — for a market where the model's languages match your callers, it is often the best experience. But for a genuinely multilingual, heavily code-switching audience, a pipeline that lets you pick the right transcription model per language usually wins. This is exactly why the ability to choose your architecture per use case matters. See building on a frozen speech-to-speech model for the broader constraints of that approach.

How to evaluate multilingual quality

Don't trust the language count on the pricing page. Test it, with audio that looks like your real calls:

• Bring accented speakers. Not clean studio English — the actual accents and regional languages your customers use.
• Force code-switching. Say a sentence that starts in one language and ends in another. Say a number in one language and a noun in another. Watch whether the transcript and the reply hold together.
• Probe the numbers. Read a phone number, an amount with currency, and a date. Check both understanding and how the agent says them back.
• Trigger a switch — and a non-switch. Confirm the agent switches when the caller genuinely changes language, and does not flap on a single foreign word.
• Check the wrong-script trap. Ask to see raw transcripts. Words in the wrong script are a sign the system isn't normalizing, and it will bite you downstream.
• Test the tail, not the mean. Average quality is easy; the callers you lose are in the tail — the strongest accents, the fastest code-switchers, the noisiest lines.

Multilingual voice AI that actually sounds local is not about supporting the most languages. It is about handling one real caller — accent, mixed languages, spoken numbers, and all — without making them slow down for the machine. For how we approach the models and languages behind this, see models and languages, and for the market where this matters most, multilingual voice agents for India.