Noisy calls, clear agents: how voice AI survives real-world (and Indian) phone audio

The short version: a voice agent survives noisy real-world calls by cleaning the inbound audio before it ever reaches speech recognition — echo cancellation first, then a denoiser tuned to isolate the foreground speaker, then voice-activity detection and endpointing thresholds set for telephony rather than a studio mic. The hard part is doing all of that in a few milliseconds per audio chunk, without over-cleaning the signal so aggressively that you chop the front off a soft "haan" or a whispered "hello." This is one of the least glamorous and most decisive subsystems in a voice platform, and it's where a lot of demos quietly fall apart the moment they meet a real Indian phone line.

This is an engineering teardown of how we do it. For the wider picture of how all these pieces fit together, see our complete guide to AI voice agents.

Why real phone audio breaks naive voice AI

A laptop demo runs on a close-talking microphone in a quiet room. A production call runs on none of those assumptions. In India specifically, the audio arriving at your server has usually been through:

• A low-bitrate carrier codec. Most PSTN audio is 8 kHz μ-law — a narrowband, lossy path that throws away everything above ~3.4 kHz. The crisp high-frequency energy that distinguishes s from f, or t from k, is simply gone before you get the bytes.
• Open-air environments. Callers pick up in markets, on the street, on a scooter, in a shared office. There's traffic, horns, wind, and crosstalk.
• Background media. A TV, YouTube, or radio playing softly behind the caller. This is the sneaky one — it's not noise, it's speech, so a recognizer happily transcribes it.
• Multiple speakers. Someone else in the room answering, a family member prompting the caller, a shopkeeper talking to a customer.
• Real echo. On many Indian PSTN routes the echo round-trip sits around 200–350 ms — long enough that the agent hears its own voice come back as if the caller said it.

Feed that raw into a speech-to-text engine and you get hallucinated turns, the agent answering the television, clipped first words, and endless false interruptions. Cleaning it up is not optional.

Echo cancellation comes first

Before any denoising, you have to remove the agent's own voice from the inbound stream. If you skip this, the agent's playback leaks back up the line and gets transcribed as user speech — the classic "the agent interrupts itself" bug.

We run a WebRTC-style acoustic echo canceller (AEC) at the telephony boundary. Two implementation details matter for Indian lines:

• Run the AEC at 16 kHz even when the carrier is 8 kHz. Upsampling the near-end audio before echo cancellation gives the adaptive filter roughly twice the effective tail length, which is what you need when the echo round-trip is 200–350 ms.
• Feed the reference paced, not in bursts. The far-end (agent) audio has to be aligned in time with the near-end (caller) audio. If the outbound side sends a ten-second greeting as one big chunk, you queue it and drain it one frame per inbound frame, so reference and echo stay temporally aligned. Misalignment here is the difference between the echo canceller working and doing nothing.

Echo handling also feeds into a later stage: after the agent stops speaking there's a short window where residual echo can still leak in, so the post-playback gate stays closed a little longer. How long depends on which denoiser path the call is on — more on that below.

Denoising: the dual-path design, and why we don't switch mid-call

Denoising is where the real tradeoffs live. There is no single denoiser that is both maximally aggressive and perfectly safe for speech, so we run two paths and pick one per call.

The slim path uses a modern, licensed speech-enhancement model (the Quail Voice Focus family from ai-coustics) whose whole job is foreground speaker isolation — it doesn't just attenuate hiss, it actively suppresses competing background speakers, which is exactly the failure mode on a crowded Indian call. It runs at roughly 6–8% of one vCPU per call at 16 kHz, and it has a confidence-style enhancement knob: conservative, balanced, or aggressive. Crucially, this model already normalizes loudness, so on the slim path we do not run a separate automatic gain control (AGC) stage afterward — a second stage chasing the same loudness target was the biggest source of double-processing artifacts we removed.

The legacy path is the fallback chain: AEC, then a DeepFilterNet denoiser, then a static gain stage, then AGC targeting roughly −20 dBFS. DeepFilterNet has its own aggressiveness dial — an attenuation limit, defaulted to about 20 dB. Push it toward 40 dB and you remove more noise but start distorting speech; drop it toward 12 dB and you keep more natural texture but leave residual noise. 20 dB is the balance point we've found for PSTN.

Why keep two? Because the modern model is a paid, licensed SDK, and a license can lapse, a payment can fail, a model file can go missing after a partial deploy, or an on-call engineer can deliberately flip everyone to the fallback. In any of those cases the call must still complete with the same audio behavior it had before. The legacy chain is a byte-for-byte preserved safety net, pinned by tests so a refactor can't silently change what it produces.

The decision is made once, at call start, and is fixed for the life of the call. This is deliberate. Switching denoisers mid-call would create an audible discontinuity in the audio stream — and the speech engines would interpret that seam as an utterance boundary, splitting a sentence or firing a false end-of-turn. A consistent (even if slightly less clean) signal beats a signal that suddenly changes character halfway through a word.

The danger of over-aggressive denoising

It's tempting to crank cleaning to the maximum. Don't. The failure mode is subtle and expensive: aggressive denoising clips the soft onsets of consonants.

Soft consonants — the leading f, s, th, or an unvoiced stop — carry very little energy. A denoiser tuned to nuke anything quiet will treat those onsets as noise and shave them off. The recognizer then hears "\_ine" instead of "fine," or drops a word boundary entirely. On narrowband PSTN audio, where the high frequencies that define those consonants are already attenuated by the codec, there's almost no margin left to give away.

This is why we're careful about stacking cleaners. When the slim path's speech-focus model is active, we turn off the recognizer's own internal noise gate — running a second voice-activity pass on already-cleaned audio is redundant work whose main effect is clipping consonant onsets. One good denoiser, tuned conservatively, beats two aggressive ones fighting each other.

VAD and endpointing: real speech peaks lower on the phone

Voice-activity detection (VAD) decides when the caller is speaking. It runs a small neural model (Silero, in our pipeline) that outputs a speech probability per frame; above a threshold, it's speech. The default threshold that works on clean mic audio is too high for the phone.

On PSTN, the speech-probability peaks are lower — real speech often tops out around 0.37–0.43 rather than pushing toward 1.0 — because the codec has stripped the energy the model was trained to key on. So we cap the effective VAD threshold on telephony calls (default cap around 0.45, base threshold around 0.3). If an operator sets an aggressive threshold that's fine for a headset, the cap quietly protects them from missing quiet phone speech. Set it too high and the agent simply won't hear a soft speaker at all.

Endpointing — deciding the caller has finished — is tuned per path too. On the cleaner slim-path signal we can wait less time in silence before declaring end-of-turn (around 0.6 s) because the noise floor is flat and a background hum won't be mistaken for continued speech. On the noisier legacy path we wait longer (around 1.0 s) so we don't chop the end off a turn when the line is crackly. Tighter endpointing is only safe on cleaner audio — which is the whole point of getting the denoising right first.

Rejecting the television: an energy floor so the agent doesn't answer a TV

Background media deserves its own defense because it defeats denoising by design — it is speech, so a speech-focused cleaner won't remove it, and a recognizer will faithfully transcribe the car review playing on YouTube behind the caller. Left unchecked, "tell me about the on-road price" arrives as a fake user turn and the agent answers the TV.

The tell is energy, not content. A real caller speaking into the handset peaks well above ~1000 RMS even when they're being quiet; leaky background media, picked up from across the room, peaks down around 100–250 RMS. So we track the peak energy of each speech session and drop transcripts whose entire session stayed below a session RMS floor (default around 250). Two exemptions keep this from backfiring:

• The greeting phase and the first user turn are exempt. A soft first "hello" or "haan" — handset mic plus room reverb — can legitimately sit below the floor, and we'd rather over-accept the opening than miss it. After the first turn, the floor is fully active.
• Only meaningful text on the first turn is exempt, so random ambient chatter still gets dropped because it usually fails a basic meaningfulness check anyway.

It's a blunt instrument on purpose. Content-based filtering of "is this the user or the TV?" is unreliable; an energy gate with careful exemptions is robust and cheap.

How noise interacts with barge-in and endpointing

Every noise-handling knob above is really a barge-in knob in disguise. Barge-in — letting the caller interrupt the agent mid-sentence — depends on the same VAD that noise corrupts. Get the thresholds wrong and you get one of two bad calls: an agent that talks over the caller because it never detected the interruption, or an agent that stops dead every time a horn honks because it thought the noise was a barge-in.

The chain of dependencies is direct: clean audio → a flatter noise floor → a VAD threshold you can trust → confident barge-in and tight endpointing. Skimp on the DSP and every downstream timing decision inherits the noise. We go deeper on the interruption side in barge-in and interruption handling in voice AI, and the same audio cleanliness is what makes reliable voicemail and answering-machine detection possible — you can't classify a beep or a greeting you didn't hear cleanly.

India-specific nuances

Most of the above is universal, but India stacks the difficulty:

• Carrier variability. Routes and quality differ wildly across operators and circles. Echo tails, codec behavior, and packet jitter aren't constant, which is part of why a fixed-at-call-start path plus per-path tuning beats one global setting.
• Accents and clarity under noise. Indian-English and regional accents already push a recognizer; add market noise and the margin shrinks. This is exactly why we favor a foreground-speaker isolation model over plain noise suppression — keeping the target speaker intelligible matters more than a silent background.
• Code-switching meets noise. Callers switch between Hindi, English, and a regional language mid-sentence. Denoising has to preserve the phonetic detail that language and script detection rely on; over-cleaning that clips consonants makes code-switch detection worse, not better. We cover the language side in multilingual voice AI: accents and code-switching.
• Louder, more crowded rooms. The background-media and multi-speaker problems are simply more common, which raises the stakes on the RMS floor and the speaker-isolation model.

The models and languages we run these paths against are listed on our models and languages page, and the broader constraint of building on top of speech models you don't control is covered in building on a frozen speech-to-speech model.

What to actually ask a vendor

"Do you have noise cancellation?" is the wrong question — everyone says yes. Ask how:

• Do you run echo cancellation, and how do you handle 200–350 ms PSTN echo tails?
• Is your denoiser tuned to isolate the foreground speaker, or does it just attenuate broadband noise?
• How do you avoid clipping soft consonants on narrowband audio?
• Is your VAD threshold tuned for telephony, where real speech peaks lower?
• What stops the agent from answering a TV playing behind the caller?
• Do you have a fallback if your denoising vendor's license lapses mid-fleet?

The answers separate a system that survives a million real Indian calls from one that only shines in a quiet demo. That gap is exactly what these teardowns are about — read the rest of our engineering field notes.