Building on a frozen speech-to-speech model: the delegation pattern behind natural voice AI

Speech-to-speech models — audio in, audio out, no separate transcription or text-to-speech step — are the most natural-sounding voice AI available. They're also the most constrained thing to build a real product on: the instructions are locked at the start of the session, you can't edit them mid-call, and the model voices whatever it decides to say with limited room for your business logic. The pattern that makes them usable in production is delegation: the live speech model handles the conversation and the voice, while a second, swappable "delegate" model does the actual thinking and tool calls. This post explains that architecture and, just as importantly, where it gets hard.

This is where a lot of voice AI is heading, so it's worth understanding. For the ground-level basics first, see the complete guide to AI voice agents.

What a speech-to-speech model gives you — and takes away

The upside is obvious the first time you hear one: natural prosody, real interruptions, no robotic TTS seams. The constraints are less obvious and they shape the entire architecture:

• A fixed, one-shot instruction budget. You get a bounded number of instruction tokens, set once at session start, with no replace. Everything the model needs to be has to fit, up front.
• No mid-call instruction edits. You can't rewrite the system prompt when the conversation changes topic.
• It voices, it doesn't render. A speech-to-speech model says text in its own voice; it can paraphrase. It is not a TTS engine playing your exact string.
• A speech-model knowledge cutoff that's often earlier than the text models you're used to.
• Concurrency ceilings — realtime sessions are capped per tier, which is a real constraint for bulk outbound.

If you tried to cram all your tools, policies, and dynamic knowledge into that frozen instruction budget, you'd run out immediately and still couldn't change anything mid-call. Hence delegation.

The delegation pattern

Split the job in two:

• The live speech model owns the audio: listening, turn-taking, interruptions, and speaking. It's the mouth and ears.
• A delegate model (any capable text LLM) owns the reasoning: deciding what to do, calling tools, extracting data, applying business logic. It's the brain.

The live model, when it needs something done, hands off to the delegate; the delegate's answer comes back and the live model voices it. Crucially, the delegate is the mutable half — you can change delegate models, prompts, and tools freely, because the frozen part (the speech model) only needs to know how to converse and when to delegate.

This also solves the instruction-budget problem elegantly: tool definitions live with the delegate, costing zero of the live model's tight instruction budget. You can wire in dozens of tools without touching the speech model's limits, and swap the delegate for any backend model you like.

Where it gets hard (the parts demos skip)

1. Don't freeze the audio loop while the brain thinks

The naive implementation awaits the delegate call inline — and freezes the receive loop for the whole LLM-plus-tool round trip. No audio, no transcripts, no barge-in until it returns. The user experiences a dead call. The fix is to run delegation as a background task so audio keeps flowing while the delegate works, then inject the result when it's ready.

2. Every delegated token is spoken — and billed as voice

Because the live model voices the delegate's output, a long delegate answer becomes minutes of speech. An 8,000-token essay from the delegate is roughly ten minutes of talking — expensive and unusable. Delegation has to be tuned to return tight answers, and you have to think about cost in spoken wall-clock time, not just tokens.

3. Send the delta, not the whole transcript

If each delegation resends the entire conversation so far, the k-th delegation carries k overlapping copies of the dialogue — cost and latency balloon. Send only what's new since the last handoff.

4. Transfers and verbatim messages need care

When the agent transfers a call, the speech model holds no tools and the delegate's text arrives after the provider transfer already fired — so the caller can hear a silence gap unless you design for it. And because the model paraphrases, any message that must be exact (a compliance disclosure, a legal script) should be rendered through a classic TTS pipeline instead. Mature platforms let you pick per use case. (Related: how we sequence tools and handoffs in orchestration, and the verbatim tradeoff in voicemail detection.)

5. Sessions expire; long calls need reconnect logic

Realtime sessions have a lifetime. Without explicit reconnect handling, a long call simply ends when the session expires. This is easy to miss in testing (short calls) and painful in production (that one 20-minute call that dies at minute 15).

Why the pattern matters to buyers

Delegation is why a platform can offer the natural sound of a speech-to-speech model and deep tool use, your choice of reasoning model, and controllable behavior. A platform that only exposes the raw speech model gives you great audio and shallow capability; one that only uses a pipeline gives you control but a more synthetic voice. The delegation architecture is how you stop choosing.

When evaluating, ask: Can I use a speech-to-speech voice and call my tools and CRM mid-call? Can I choose the reasoning model independently of the voice? What happens on long calls, transfers, and messages that must be exact? The quality of those answers tells you whether a team has actually shipped this or just demoed it.

More on how we approach model and channel choice in models & languages, and how to weigh all of this when choosing a voice AI agent.