How to write effective prompts for AI voice agents

By the end of this guide you'll be able to write a prompt for an AI voice agent that produces short, speakable turns, looks facts up instead of guessing, ends calls cleanly, and can be changed without breaking what already works. The core idea: a voice prompt is a behaviour spec. It isn't a script, and it isn't where your facts live.

Most prompt advice is written for chat. Three things change on a phone call:

• Everything the model writes is spoken. A list, heading or stray label becomes audio.
• Every token costs time on every turn. The prompt is re-read before each reply, and callers notice a slow first word immediately.
• Your prompt isn't the only one. The platform wraps it with its own rules for speech, ending calls, saving data and switching languages. Contradict them and the agent turns inconsistent.

New to building agents? Start with how to set up an AI voice agent.

Before you start

• One narrow job with a written definition of done ("a slot is booked, or the caller declines").
• Ten to twenty real transcripts or recordings of your team handling this call.
• The agent's tools, split into lookups (read data) and actions (book, save, change).
• The per-call variables you'll have, each with a sensible default.
• The voice you'll use: language, accent and gender.

Step 1: Write the job in three lines

Identity, goal, definition of done. If they don't fit in three lines, the job is too broad for one agent.

You are Asha, calling from the clinic's front desk. Your goal is to reschedule the caller's missed appointment. The call is done when a new slot is booked or the caller says they don't want one.

Then write goals as short, separate lines. Callaro's builder takes them as a numbered list with a length cap on each: "Confirm identity", "Ask why they missed it", "Offer two slots", "Book the slot the caller picks".

Step 2: Give every rule one home

Put each instruction in exactly one labelled section, so both the model and you can find it when a call goes wrong. Callaro's prompt builder is organised this way: Identity, Knowledge base, Call control and flow, Objection handling, Discovery and qualification, Close and next steps, and Guardrails, followed by custom instructions and a style addendum.

• Tone and persona go in Identity, once, not sprinkled through every section.
• Objections get the caller's actual words from your transcripts and a short reply for each.
• What the agent must never do goes in Guardrails. Our guardrails guide covers writing and testing those.
• Rules every agent shares, such as a recording disclosure, belong in one place. Callaro can apply a tenant-wide "platform rules" pack before each agent's own sections.

Step 3: Keep facts out of the prompt

What's left in a good voice prompt is behaviour. Facts live elsewhere:

1. Variables for anything that changes per call, written as placeholders such as {{name}}. Give each a default: in Callaro a missing contact field renders empty, so "Hi {{name}}, this is Asha" becomes "Hi , this is Asha" unless you set a fallback.
2. The knowledge base for reference material: policies, FAQs, opening hours.
3. Tools for anything live: availability, order status, balances.

This helps speed as well as accuracy. On models that support prompt caching, Callaro caches the static part of an agent's prompt and fills per-call placeholders separately, so one cached prompt serves every contact. Facts pasted into the prompt text are re-read every turn and go stale the day something changes.

Then say where facts may come from: "Only state dates, availability or account details that came from a tool result or the knowledge base in this call. If you don't have it, check, or say you'll find out." Without that line, a model will happily answer from training data that may be years old.

Step 4: Write for the ear

Phone turns should be one or two short sentences, one question at a time. "Keep answers short" barely moves output length; a structural rule does ("At most two sentences, then wait for the caller"), especially when the examples in Step 5 obey it.

• No lists, headings or labels in replies. "Options:" followed by three items sounds like a brochure read aloud.
• Leave number formatting to the platform, but specify readback. Digits versus words depends on the text-to-speech engine, and on Callaro's pipeline the platform adds rules for the engine you picked. What you add: read phone numbers back in small groups, and confirm a date before booking it.
• Punctuation is prosody. A comma is a short pause, a full stop resets intonation, a question mark lifts it. Exclamation marks make expressive voices over-act.
• Match the persona to the voice. In Hindi, Marathi and many other languages, verbs change with the speaker's gender. A male voice saying feminine forms ("main check karti hoon") is an instant tell.

Step 5: Show the register with examples

Examples are your strongest lever, because the model copies them. Write ten to twenty-five short exchanges from real transcripts: the happy path, a busy caller, "are you a bot?", an off-topic question, a clear no, and the close. For multilingual agents, include examples in each language, in the script your voice reads best (more in our post on accents and code-switching).

Two cautions from production:

• An example beats a rule, in both directions. We've seen a rule written as a labelled template ("Spoken reply: … Tool call: …") come out of the model as text, and on a call, text gets spoken. Describe behaviour in prose; only show lines the caller should hear.
• Vary the examples. If every example opens with "Sure!", every turn will too.

Step 6: Put tool guidance where the model reads it

The agent answers instead of looking it up. We've seen fast, low-reasoning models skip a lookup and answer from memory despite a rule against it. Models use the middle of long inputs less reliably than the start and end (Liu et al.), so put the trigger in the tool's own description: "Call this before you state a slot or confirm availability."

The agent stalls out loud. Don't script "let me check that for you" before every tool. Callaro plays a short "one moment" in the caller's language only while an external request is actually in flight, and answers a repeated read-only lookup with the same arguments from the result the call already has. A scripted stall line adds a pause even when the answer is instant.

The agent invents values for actions, classically a made-up phone number written into a CRM. Don't ask the model for values you already know. In Callaro a tool parameter can take its value from a static setting, a call variable such as the caller's number, a captured field or a previous tool's output; config and variable values are hidden from the model and always win. For values the model does supply, Callaro can check that a phone number or email was actually said or already known, and send the agent back to ask if not.

Step 7: Don't fight the platform's rules

Before adding a rule, check whether the platform already owns that behaviour. On Callaro, the platform speaks the greeting, then tells the model what the caller heard and not to re-greet mid-call. A normal call ends only when the caller says goodbye or confirms they need nothing more, with the closing line and hang-up in one response. Captured data is saved silently, and language switches stay within the languages you allow.

The conflicts we see most, and what they cause:

• "Confirm every detail after you save it" doubles confirmations and slows the call. Ask for readback where it matters, before saving.
• "End the call once the appointment is booked" either gets ignored or hangs up on a caller with one more question.
• "Always greet the caller warmly" invites re-greetings mid-call.
• "Always reply in Hindi", on an agent that also allows English, strands an English-speaking caller.

Don't copy the platform's formatting rules into your prompt either: they're updated centrally, and a copy becomes a contradiction after the next update.

Step 8: Test it like code

1. Call it with real data. Callaro's test call dials through one of your numbers and can pull a CRM contact, so variables fill as in production. Share links let colleagues test, with an expiry, a use cap and locked variables.
2. Run a break-it script before every release: interruptions, silence, an off-topic question, a request for a human, a language switch.
3. Change one thing at a time. Every save and publish creates a version you can restore.
4. Measure at volume. Run a campaign A/B test across two agent variants; Callaro pins each contact to one variant across retries and snapshots the version at launch. Score every call with a post-call analysis prompt ("Did the agent state a date that didn't come from a tool?") instead of reading a sample.
5. Watch for long turns. The live call monitor shows lines the agent prepared but never said because the caller cut in. Many of those means turns are too long.

Gotchas from real deployments

• Prompt creep. Every bad call earns a new rule until the prompt is long, slow and self-contradicting. When you add a line, remove or merge one.
• Control words that get spoken. "STOP and wait" has come out of a model as the spoken word "stop". Write "then wait for the caller".
• Memory treated as fact. If the agent keeps notes from earlier calls, last call's wrong answer can return as this call's "fact". State that history is context, not a source of facts.
• The script tax. Devanagari and other non-Latin scripts usually take more tokens than the same English content, by an amount that depends on the tokenizer (Srivastava, 2026). Consider English for instructions and the caller's language for examples. Callaro's builder shows a running token estimate, counted by script for full-duplex speech models.

Your prompt checklist

• [ ] Identity, goal and definition of done in three lines; goals as a short list
• [ ] Every rule in exactly one labelled section; shared rules in one shared pack
• [ ] Per-call facts in variables (with defaults), the knowledge base or tools, not in the prompt text
• [ ] One line saying where facts may come from
• [ ] "At most two sentences, one question per turn", with examples that obey it
• [ ] Readback rules for phone numbers and dates
• [ ] Persona, voice gender and example grammar agree
• [ ] Ten to twenty-five real examples in every allowed language, no labelled templates
• [ ] A "call this before…" trigger in each lookup tool's description
• [ ] Known values bound to tool parameters, not left to the model
• [ ] Nothing that contradicts the platform's greeting, ending, saving or language rules
• [ ] Break-it script passes; one change per version; A/B test and post-call scoring before you scale

Common questions

How long should a voice agent prompt be?

As short as the job allows. Signs it's too long: wordier turns, ignored rules and a slower first word. Moving reference material into the knowledge base usually removes most of the length.

Do speech-to-speech models need a different prompt?

The principles hold, but the limits are tighter: full-duplex speech models take their instructions once, at session start, within a fixed budget. We cover designing around that in building on a frozen speech-to-speech model.

Next step

Pick one agent, read five recent transcripts and find the longest turn. Fix the rule or example behind it, re-run your break-it script, and ship a new version. For the bigger picture, read the complete guide to AI voice agents, or see how prompts, tools and handoffs fit together in Callaro's orchestration layer.