How AI voice agents detect voicemail (and why it's harder than it looks)

Answering-machine detection (AMD) is the unglamorous subsystem that decides whether a real person picked up the phone or whether your agent is about to deliver a perfect pitch to a voicemail greeting. Get it right and your outbound program runs clean. Get it wrong and you either talk over people's greetings, burn paid minutes on dial tones, or hang up on real customers mid-"hello." This is one of the hardest problems in real-time voice AI, and almost nobody talks about it honestly.

This post is a practical walkthrough of how a production voice agent actually makes that decision — the signals it uses, the two-stage nature of the problem, and the edge cases that quietly wreck naive implementations.

If you're evaluating platforms, this is the kind of depth to probe for. For the broader picture, see our complete guide to AI voice agents.

The naive approach — and why it fails

The tempting first implementation is: "listen for a beep, or ask the carrier if it's a machine, then leave a message." Both halves are individually unreliable.

Carrier-side AMD (Twilio's AsyncAMD, Exotel's AMD passthrough, and equivalents) is a probabilistic classifier running on the telco's side. It's useful but it's a guess, it arrives with latency, and its accuracy varies wildly by region and carrier. Runtime audio detection — classifying the first few seconds of audio yourself — is faster to react to but easy to fool with a chatty human who opens with a long monologue, or a voicemail greeting that sounds conversational.

The robust answer is to treat AMD as a fusion problem: combine carrier signals and your own runtime audio classification, and design a state machine that survives either source being wrong or late.

Voicemail detection is two problems, not one

The insight that separates production systems from demos: detecting that you've reached a voicemail is only half the job. The other half is knowing when to speak.

A voicemail interaction has a specific shape:

1. The machine greeting plays ("Hi, you've reached… I can't take your call…").
2. Then there's a record cue — the beep, often preceded by "at the tone, please record your message."
3. Only after the beep will the mailbox actually capture audio.

If your agent detects "this is voicemail" and immediately starts talking, it talks over the greeting and the first half of your message is lost — recorded before the beep, or clipped. So a real system runs a small state machine:

• awaiting signal — no decision yet; the agent behaves normally.
• machine detected, awaiting beep — we're confident it's a machine, so we suppress the agent and wait.
• beep / record cue detected — now, and only now, deliver the message.
• terminal states — hung up, message left, ignored, or "no signal" (treat as human).

That "wait for the beep before speaking" step is what makes voicemail messages land cleanly instead of arriving as "—and that's why you should call us back."

Fusing carrier and runtime signals

Each signal source has a different latency/accuracy tradeoff, so a good system records which source fired and when. In practice you'll see events like: the carrier classifier decides "machine" at t=2.0s, and the runtime audio classifier corroborates at t=3.4s. Logging both — with offsets — is invaluable for tuning, because it lets you answer "did we hang up because we knew it was voicemail, or for some other reason?" after the fact.

A few hard-won principles:

• Machine-indicator phrases ("the person you are trying to reach," "please leave a message after the tone") are strong runtime signals — but they should mark the call as voicemail and move to awaiting beep, not trigger immediate speech.
• Region matters. Carrier greetings differ by market. Phrase packs mined from real production transcripts for specific carriers dramatically improve runtime accuracy versus a generic English word list.
• Grace windows are per-region. Some carriers will drop the call or ask you to re-record if you sit silent too long waiting for a beep. So the "how long do I wait" timeout has to be tuned per market, not set to one global value.

The race condition nobody warns you about

Here's where naive implementations get subtly broken. Two signals can arrive nearly simultaneously — the carrier says "machine" while the runtime classifier is mid-decision. If both independently trigger the "leave a message" side effects (hang up, play the message, enqueue a follow-up), you get double actions: two follow-ups queued, or a hangup racing with playback.

The fix is disciplined concurrency. The decision — and all of its side effects — must run under a single lock, with a "has this already fired?" check inside the lock. A fast path can peek before locking to avoid overhead, but the actual decision and its side effects serialize: the first signal wins, the second sees the decision already made and returns quietly. It sounds obvious written down; it is routinely gotten wrong, and the symptom (occasional duplicate voicemails or dropped calls) is maddening to debug because it only happens when timing lines up.

There's a related teardown edge case: once you've decided to end a voicemail call, the speech-to-text loop can keep transcribing the dying call and "evaluating" audio that no longer matters — which, in one real incident, produced a false language switch in the last second of a call. The lesson: gate post-termination processing explicitly.

Voicemail is even harder with speech-to-speech models

The newest voice models are speech-to-speech — audio in, audio out, no separate text-to-speech step. They're wonderfully natural, but they complicate voicemail in two ways.

First, you can't cleanly "mute" the model while waiting for the beep. If you gate the model's input to stop it talking, you also stop it producing the input transcripts that feed your beep classifier — so you'd never detect the record cue. The muting and the detection are coupled.

Second, a speech-to-speech model voices your message rather than reading it verbatim; it can paraphrase. For most calls that's fine. But if your voicemail message must be legally or commercially exact (a compliance disclosure, a specific offer), you want a pipeline that renders the text through classic text-to-speech, where "what you wrote is what plays" is guaranteed. This is a genuine architectural tradeoff, and a mature platform lets you choose per use case. (More on how we orchestrate models and tools mid-call in how our voice agents take real actions.)

Why this matters for your outbound program

AMD quality shows up directly in your numbers:

• Cost. Every second spent talking to a machine you didn't detect is a paid minute wasted, at scale.
• Answer quality. Clean beep detection means the voicemails you do leave are complete and professional, which lifts callback rates.
• Compliance. Knowing you're on a machine (and being able to guarantee verbatim messages) matters where disclosures are regulated.

When you evaluate a voice AI platform, don't accept "yes, we do voicemail detection" at face value. Ask how: Do they fuse carrier and runtime signals? Do they wait for the beep, or just start talking? How do they handle the race between two signals? Do they tune grace windows by region? The answers separate a demo from a system that survives a million real calls.

That depth is the whole reason we write these teardowns — see the rest of our engineering field notes or read how we approach choosing a voice AI agent.