Speech-to-speech vs pipeline voice AI: choose per call, not per platform

Speech-to-speech or pipeline? As of October 2026 the right answer is both, chosen per call type. Latency used to settle the question, and it no longer does. In the eight days to 5 October, every stage of the pipeline gained a faster model, and the newest speech-to-speech systems stopped making callers wait while the model thinks.

What separates the two paths now is control. A pipeline passes text between its stages, and text can be checked, gated, timed and replaced. A speech-to-speech model passes audio, which sounds more natural and leaves far fewer places to step in. We run both in production. Below: what changed, what each path cannot do, and a rule for choosing.

If the terms are new: a pipeline (or cascade) chains speech-to-text, a language model and text-to-speech. A speech-to-speech model takes audio in and gives audio out. The complete guide to AI voice agents covers the basics.

What changed between 28 September and 5 October

Five announcements in eight days, pointing the same way from both sides.

Every pipeline stage got a faster option

• Text-to-speech. ElevenLabs released Eleven v4 and a low-latency v4 Turbo on 28 September, reporting a median of roughly 150 ms to first speech for Turbo.
• The language model. Inception made Mercury Voice, a diffusion language model built for voice agents, generally available on 29 September. Its company-reported median is 320 ms to the first answer token.
• Speech-to-text. Microsoft released MAI-Transcribe-2-Streaming on 1 October, with first words reported just over 100 ms after audio starts. Launch coverage quotes Microsoft's documentation describing it as a preview with no service-level agreement.

All three figures are vendor-reported, and each covers one stage. Telephony, network hops and endpointing (deciding the caller has finished) sit on top. What they do show is the stage-by-stage penalty for running a pipeline shrinking.

Speech-to-speech stopped blocking on the thinking

OpenAI set the pattern on 10 September with GPT-Live-1, which holds the conversation and hands reasoning to a separate backend model. We wrote about it at launch. Two more releases followed within days of each other.

• Decagon's Voice 3 (announced on 1 October) runs two layers in parallel: a low-latency conversational model that listens and speaks, and a more powerful one for reasoning, tool calls and guardrails.
• Amazon's Nova 2.5 Sonic reached general availability in Bedrock on 5 October. Its feature list includes asynchronous tool calling, so a tool can run without the conversation stopping.

Both camps now have the same shape: something fast that talks and something slower that thinks, side by side. We covered that design and where it breaks in building on a frozen speech-to-speech model.

What a public benchmark says, read carefully

Bluejay's MIVAS leaderboard is a public test that scores voice agents on finished work, not on how they sound. Simulated callers ring a multi-agent system in healthcare, legal and customer support. A call passes only if its tool calls, handoffs and final database state are all correct, and every task runs five times.

The board published on 26 September shows three things.

• The top entry, GPT-Live-1 delegating to a current reasoning model, passes 87.8% of tasks on a single run and 66.3% when a task must pass all five runs, at a median 1.81 seconds.
• The cascaded baseline passes 35.8% and 24.5%, at 3.10 seconds.
• Two smaller speech-to-speech models score below that baseline on the all-five measure, and one of them is slower than it.

Three readings matter more than the ranking.

1. The thinker decides whether the job gets done. The top two rows share one voice layer and differ only in the reasoning model behind it. Another model gains twelve points on the all-five measure when extended thinking is switched on, at almost the same median latency.
2. The baseline is no verdict on pipelines. Its language model is GPT-4.1, an earlier generation than the reasoners behind the leading rows. No row pairs a pipeline with the leader's reasoning model, so the board can't say which audio path wins on equal brains.
3. Consistency is the real gap. Even for the leader, about one task in three fails at least one of its five runs. Ask a vendor for the all-five number first.

The companies are fictional, the callers are simulated, and Bluejay sells voice-agent testing. Treat the board as a method worth copying more than as a league table.

What you can control depends on what's in the middle

If both designs have a talker and a thinker, what differs is the talker. In a pipeline it is three parts with text passing between them. In speech-to-speech it is one model with audio at both ends.

When we added a full-duplex speech-to-speech path beside our pipeline in September, we audited 105 pipeline behaviours against it. About half were at parity. Nineteen could not exist at all and needed a different mechanism. The rest needed fixing or rebuilding.

The nineteen trace back chiefly to two missing parts: a recogniser you can put a gate in front of, and a speech engine you can hand an exact string to. The differences a caller can hear are below.

Exact wording

On the pipeline, the lines the platform owns (the greeting, a fixed voicemail message, the transfer announcement) go to the speech engine as written. A speech-to-speech model has no such engine. The platform can only instruct it to say a line, and OpenAI's session guide notes that the model may paraphrase what it is given to say.

So a sentence that must be exact belongs in a platform-spoken line on a pipeline agent. Anything a language model composes mid-call isn't guaranteed word for word on either path.

Evidence about what was heard

A recogniser reports more than words. Where it supplies a confidence score for each turn, our transfer gate uses it: a transfer attempted on a low-confidence transcript is held while the agent asks one yes-or-no question. Speech-to-speech engines report no such score, so only the simpler rules apply there, such as no transfer on the caller's first turn.

Gating what the model hears

In a pipeline, voice-activity detection, a noise gate and an echo gate sit in front of the recogniser. A speech-to-speech model detects turns on the vendor's servers, so there is nothing to gate. Our echo check on that path had to move to the text of the transcript. Line-level noise suppression still runs for both.

One consequence surprised us. When an outbound call reaches voicemail, the agent should stay silent until the beep. A pipeline simply doesn't generate. A speech-to-speech model can't be muted the same way, because muting the audio we send it also stops the transcripts our beep detection reads. What's left is an instruction to stay quiet, with nudges and backend calls suppressed.

Stopping mid-sentence

Full duplex means the model listens while it speaks. It does not clear speech already queued toward the phone line, and these models generate audio in bursts, so some is always waiting in the playout buffer.

The API we use documents no event for a caller starting to speak, so the first signal is the caller's transcript. On our first test calls that arrived more than a second after they began talking.

We built a local voice-activity pause to bridge the gap. It made greetings stutter whenever the transcript came later than the pause allowed, so it ships switched off. On the pipeline the platform owns the interruption: a pause is armed when speech starts, and audio is cut only once sustained speech is confirmed. A quick "mm-hmm" passes.

Delivery, and language

Pipeline speech engines take direction: emotion cues on some, and text written the way numbers and identifiers should be read. Our platform writes those rules per engine. The full-duplex model's documented audio setting is the choice of voice. Pace and tone become prompt guidance, and in our experience that drifts over a long call.

Language is the sharpest case. That model's voice list is English and Portuguese, and OpenAI advises testing a voice with the languages you need. For Hindi or Marathi we run that listening test first. So far a pipeline with a voice built for the language has sounded more local to us.

Seeing where the time went

A pipeline turn can be timed stage by stage: the final transcript, the language model's first token, the first audible speech, the time spent in tools. A speech-to-speech turn gives one number, voice to voice.

Even that number is easy to get wrong. Vendor transcripts lag the audio, by one to three seconds on our test calls, so timing a reply from the transcript made some turns look almost instant. We now time from the end of the caller's voice, detected on our own side. When a vendor quotes speech-to-speech latency, ask where the clock starts.

Failing over

Three stages give three places to recover. On our pipeline, a rate-limited language-model call can be re-sent to a fallback model before any audio has gone out. One of our speech integrations hands a failed line to a second engine, when one is configured, so the caller still hears the sentence.

A speech-to-speech call is one session with one vendor. Sessions have a duration limit and accounts have a ceiling on concurrent sessions, which is worth checking before a bulk campaign.

What the pipeline gives up

Text carries words and nothing else, so tone, hesitation and a half-finished thought are flattened at the recogniser. A pipeline also has to decide the caller has finished before it starts thinking. No faster model removes that wait. It can only be tuned, or gambled on with speculative replies.

During a lookup our pipeline plays a short hold line. A full-duplex model can keep talking with the caller while the backend works.

How to choose, call type by call type

Ask these in order and stop at the first yes.

1. Must any sentence be word for word? A disclosure, a recorded-line notice, a voicemail message. Use the pipeline, with that sentence in a platform-spoken line.
2. Is the call in a language the speech-to-speech voice wasn't built for? Listen first, and default to a pipeline with a native voice.
3. Does a campaign need more simultaneous calls than your session ceiling allows? Use the pipeline, or split the list.
4. Will a regulator or an enterprise client ask you to explain a slow or wrong turn stage by stage? Use the pipeline.
5. Is the call won on feel? Discovery, objection handling, callers who interrupt. Use speech-to-speech with delegated reasoning.

If none of the first four applies, speech-to-speech is where we'd start for conversations in a language it speaks natively. Tool-heavy calls can go either way, provided tools run without freezing the audio and the agent passes the check below.

A worked example

Take a lender with three call types on one platform.

• Payment reminders with a mandatory disclosure, in Hindi and Marathi. Pipeline. Rules 1 and 2 both apply.
• Inbound English support with account lookups. Speech-to-speech, with lookups delegated to a reasoning model and transfers to a person restricted to an allowlist.
• A seasonal outbound campaign across a large list. Pipeline, because concurrency and voicemail handling decide the outcome.

That is one agent builder and three configurations. On Callaro the conversation mode and the models are chosen per agent.

Test it yourself: the all-five check

The MIVAS method transfers to your own calls.

1. Pick twenty real tasks, each with a checkable end state such as a booking row, a ticket or a field in your CRM.
2. Write each one as a brief a colleague or a scripted caller can run.
3. Run every task five times on each path.
4. Score the system state, not the transcript. A pleasant call that booked the wrong slot fails.
5. Count a task only if all five runs pass, and note the median time to first audio, measured from the end of the caller's speech.
6. Work out cost per completed task: connected minutes × the rates metered on that path, plus reasoning-model usage, divided by the tasks that passed.

The two paths are metered differently. A pipeline meters speech-to-text, the language model, text-to-speech and telephony separately. Speech-to-speech vendors meter the voice layer by connected time or by audio tokens, with the reasoning model on top. Ask how silence is counted, because a speech-to-speech session with no voice-activity filter in front of it receives the silences too. Put your own numbers in from current pricing.

Where neither path is ready

Two limits apply whichever you pick.

• Unattended multi-step writes. The best public result still has one task in three failing at least once in five runs. A workflow that writes to a system of record needs someone to catch a miss. Keep a guarded transfer, and reconcile what the agent wrote.
• Launch-week numbers. Several of these releases are previews or carry vendor-only figures, and a faster launch chart still has to be heard on your own lines. Our post on model migrations and sunsets covers swapping a model in without rebuilding the agent.

What this means if you're buying

Stop asking a vendor which architecture they use. Ask three things. Can I choose the path per agent? Which of the controls above do I lose on each? What is your all-five pass rate on my tasks?

A platform with one path will tell you that path suits every call. Bring two call types, one where wording matters and one where feel matters, and listen to both on the same script. Our options for each path are on the models and languages page.