Building multilingual voice agents for India: Hindi, Hinglish and code-switching

Building a voice agent that works in India is not the same as translating an English agent into Hindi. The Indian caller does not speak in tidy, single-language sentences. A customer confirming a delivery might say "haan bhaiya address same hai, but timing thoda evening kar dena." A borrower explaining a delay says "salary aane wali hai, main next week pay kar dunga for sure." This constant switching between Hindi and English, often within a single clause, is not an edge case in India. It is the default register of everyday urban and semi-urban speech, and it is exactly where most voice stacks fall apart.

The reason is architectural. Many speech systems are built around the assumption that one call is in one language, which you select up front. Feed such a system a naturally mixed sentence and it either forces everything into one language and mistranscribes the rest, or it flip-flops confusingly. The customer notices immediately. They hear an agent that cannot follow how they actually talk, and they either switch to terse one-word answers or hang up. In a market where trust is everything, sounding foreign is fatal.

This piece goes deep on building voice agents that genuinely work across Hindi, Hinglish, and India's broader language landscape. It covers why code-switching breaks weak stacks, the difference between telephony and studio audio, how to choose speech-to-text and text-to-speech for Indian conditions, the specific hard problems of numbers, names, and amounts, language detection and mirroring, regional voice selection, testing with native speakers, and how to scale across languages without rebuilding everything each time.

Why code-switching breaks weak stacks

Code-switching is the practice of moving between two languages within a conversation or even a sentence, and Indian speech is saturated with it. English nouns and verbs get dropped into Hindi grammatical frames constantly: "payment," "link," "account," "EMI," "balance," "delivery," "cancel." A speech-to-text system trained and configured for pure Hindi will stumble on the English words; one configured for pure English will butcher the Hindi. Neither produces a clean transcript, and everything downstream depends on that transcript.

The failure cascades. If the transcription is wrong, the language model reasons over garbled input and makes wrong decisions. If the model then replies in the wrong language or an awkward mix, the text-to-speech mispronounces it. The customer experiences an agent that misheard them and answered oddly. A stack that handles code-switching natively, by contrast, transcribes the mixed utterance faithfully, reasons over it correctly, and replies in a matching natural mix. The difference is night and day, and it is invisible in an English-only demo, which is why you must test it deliberately.

• Weak stack symptom: the agent transcribes "main kal payment kar dunga" as something garbled and asks the customer to repeat, or answers in stiff textbook Hindi that ignores the English words the customer used.
• Strong stack behavior: the agent understands the mixed sentence on the first try and replies in a natural mix that matches how the customer speaks.

Telephony audio is not studio audio

There is a second trap that is easy to miss in demos: the audio you test on is often nothing like the audio you will get in production. Demos are frequently recorded or streamed over clean, wide-band connections in quiet rooms. Real Indian phone calls arrive over narrow-band telephony, compressed, with mobile network artifacts, patchy signal, and the ambient reality of Indian life in the background: traffic, a TV, family conversation, a shop counter, wind on a two-wheeler.

Speech-to-text that scores beautifully on clean studio audio can degrade sharply on compressed telephony audio, and the degradation is worse for accented and code-switched speech. This is why you must evaluate any stack on real phone calls over your actual telephony, not on uploaded high-quality clips. An agent that transcribes flawlessly in the demo and then mishears every third word on a real mobile call in a noisy market is worse than useless, because it fails precisely where your customers are.

The practical implication is to bias toward speech-to-text that is specifically robust on Indian telephony conditions, and to test the whole pipeline end to end over the phone. Bring-your-own-telephony flexibility helps here too, because number reputation and carrier quality affect both connect rates and audio clarity. The best transcription in the world cannot recover words that the carrier dropped.

Choosing speech-to-text and text-to-speech for India

Because the platform lets you choose components per agent, you can and should pick speech-to-text and text-to-speech that are strong specifically for Indian languages and conditions, rather than accepting a generic default. These are two separate decisions with different priorities.

Speech-to-text priorities

• Accuracy on code-switched Hindi-English, not just clean single-language speech.
• Robustness on compressed, noisy telephony audio, which is what you will actually receive.
• Good handling of Indian accents across regions, which vary widely.
• Low latency, because the customer is waiting in real time and delay makes the agent feel sluggish and unnatural.

Text-to-speech priorities

• Natural, warm voice quality that does not sound robotic, because trust depends on it, especially in BFSI and collections.
• Correct pronunciation of Indian names, places, and the English words embedded in Hindi speech.
• The ability to speak a naturally mixed Hinglish reply fluently, not just pure Hindi or pure English.
• A voice that matches the region and audience, which we return to below.

Because text-to-speech is usually the largest cost component, you also balance quality against price here. A premium voice for a trust-sensitive collections or retention call is worth it; a slightly more economical but still natural voice may be fine for a high-volume reminder. The freedom to choose per agent is what makes this optimization possible.

The hard problems: numbers, names, and amounts

The single most common place multilingual voice agents fail on real Indian calls is not conversation, it is data: phone numbers, order IDs, OTPs, names, and money amounts. These are exactly the pieces that must be perfect, because a collections call that captures the wrong amount or a delivery call that mishears the address has done damage, not work.

Numbers are hard because Indians say them in mixed ways: a phone number might come as a rapid mix of Hindi and English digits, sometimes grouped, sometimes not, sometimes with "double" or "triple" for repeats. Amounts carry the lakh and crore system, which a naive system trained on thousands and millions gets wrong: "do lakh pachaas hazaar" must become the right figure, not a mistranslation. Names are hard because Indian names are diverse and easily mangled by a system tuned to Western names.

• Numbers: handle digit-by-digit capture, mixed-language digits, grouped speech, and "double/triple" constructions. Always read critical numbers back to the customer to confirm.
• Amounts: understand the lakh and crore system natively and represent money correctly; confirm the amount verbally on anything financial.
• Names and addresses: pronounce Indian names correctly and confirm spellings or key details when accuracy matters, such as delivery addresses.
• Readback as a habit: for any number, amount, or address that drives an action, the agent should repeat it back and get a yes before proceeding. This single discipline prevents most costly errors.

Language detection and mirroring

A good Indian voice agent does not force the customer to pick a language from a menu. It detects the language the customer is using and mirrors it, including the mix. If the customer opens in Hindi, the agent continues in Hindi. If they switch to English mid-call, the agent follows. If they speak Hinglish, the agent speaks Hinglish back. This mirroring is what makes the interaction feel natural rather than like filling a form.

Mirroring should be dynamic within a single call, not decided once at the start. Customers switch languages depending on the topic, their comfort, and who is nearby. An agent that locks to the opening language and refuses to move feels rigid; one that flows with the customer feels human. The design principle is simple: meet the customer where they are, in the language and register they chose, and never make them adapt to the machine.

There is a judgment call in how far to mirror. For a formal BFSI communication, you may want the agent to keep a professional register even while matching the language, rather than mirroring very casual slang. The goal is to be natural and respectful, not to imitate the customer so closely that a serious financial call starts to sound flippant. Tune this to the use case.

Choosing the right regional voice

India is not linguistically uniform, and a voice that lands well in one region can feel off in another. Beyond the choice of language itself, the accent, warmth, and gender of the voice all affect how the customer receives the call. A voice that sounds local and familiar builds trust faster than a technically correct but regionally neutral or foreign-sounding one.

Where you operate across multiple regions, consider matching the voice to the audience rather than using one voice for the whole country. A borrower in one state may respond better to a voice and accent that feels native to their region. This is not always necessary, and a well-chosen neutral Hindi or Hinglish voice covers a lot of ground, but for sensitive or high-value calls the regional fit is a lever worth using. Because voices are chosen per agent, you can run region-specific agents off the same underlying flow.

Testing with native speakers

You cannot validate a multilingual voice agent with English-speaking product managers listening to demos. The only reliable test is native speakers of each target language putting the agent through realistic calls, including the messy ones. They will instantly catch things a non-native ear misses: an unnatural word choice, a mispronounced name, a stiff mix that no real person would use, a tone that feels subtly wrong for the context.

• Recruit native testers per language and region, not one bilingual person for everything.
• Give them real scenarios and personas: cooperative, confused, angry, fast-talking, heavy code-switcher, background noise.
• Have them test over real telephony on real mobile networks, not clean connections.
• Ask them to grade naturalness and comprehension separately: an agent can understand correctly but still sound unnatural, and both matter.
• Feed findings back into prompts, voice choice, and readback rules, then re-test. Treat it as iterative, not a one-time sign-off.

Scaling across languages

Once you have one language working well, the temptation is to spin up a completely separate agent for each new language, duplicating all the logic. That path multiplies your maintenance burden and lets the versions drift apart. A better approach separates the conversation logic and the business flow, which are largely language-independent, from the language-specific layer of prompts, voice, and pronunciation rules.

With that separation, adding a language becomes a matter of configuring the language layer rather than rebuilding the whole agent. The flow, the tool calls, the escalation rules, and the compliance logic are shared; only the linguistic surface changes. This is how you get to genuine breadth, dozens of languages, without the quality of any one collapsing under maintenance debt. It also means an improvement to the underlying flow benefits every language at once.

• Separate the language-independent flow and business logic from the language-specific surface.
• Reuse compliance, escalation, and tool-calling across all languages so they stay consistent.
• Add a language by configuring prompts, voice, and readback rules, not by cloning the whole agent.
• Validate each new language with native testers before it goes live, using the same scenario set.

Takeaways

Multilingual voice for India is really about code-switching, and code-switching is where weak stacks break silently in ways an English demo will never reveal. Test on real telephony audio with real mixed-language sentences, choose speech-to-text and text-to-speech that are strong for Indian conditions, and obsess over the hard data points: numbers, amounts, and names, with readback as a standing discipline.

Detect and mirror the customer's language dynamically rather than forcing a menu choice, match the voice to the region where it matters, and validate everything with native speakers over real phone calls. Build the flow so the language-independent logic is shared and only the linguistic surface changes per language, and you can scale across dozens of Indian languages without any of them feeling like an afterthought. Get this right and your agent stops sounding foreign and starts sounding like it belongs.