It is not a full architecture doc. It is more of a map of the words that kept coming up while building the bot.

Call Flow

To give a bit of context, below is the high-level flow that depicts the moving parts of a person calling an AI Voice Agent:

1. Inbound call — the caller dials a Twilio number over the PSTN. Twilio answers and holds the call.
2. Webhook — Twilio sends an HTTP POST to the FastAPI /voice endpoint to ask what to do with the call.
3. TwiML response — FastAPI replies with TwiML containing a <Connect><Stream> instruction, telling Twilio to open a media stream to the app.
4. WebSocket + audio — Twilio opens a WebSocket to /ws and starts streaming the caller’s audio as mu-law frames. Pipecat takes ownership of the connection.
5. Audio → STT — Pipecat passes the raw audio to the STT service (e.g. Cartesia Ink). VAD detects end-of-turn before the audio is sent.
6. Transcript → LLM — STT returns a text transcript. Pipecat appends it to the conversation and sends the full context to the LLM (e.g. Gemini).
7. Response text → TTS — the LLM streams its response back to Pipecat, which forwards the text to the TTS service (e.g. Cartesia Sonic).
8. Audio → Pipecat — TTS converts the text to speech and returns audio frames to Pipecat.
9. Audio stream → Twilio — Pipecat sends the synthesised audio back over the WebSocket as mu-law media messages.
10. Audio → caller — Twilio plays the audio to the caller over the PSTN. The caller hears the agent’s voice and the loop repeats for their next utterance.

So, without further ado, I present to you…

The ABCs of AI Voice Agents

AgentCore

AWS Bedrock AgentCore is a fully managed, enterprise-grade platform that helps developers build, deploy, and scale AI agents, and it was considered as a possible runtime.

The problem, however, was that Twilio couldn’t (or I couldn’t make it) connect to it directly. Twilio Media Streams has a specific WebSocket handshake and message format. AgentCore has its own auth and protocol expectations.

Relevant docs: AWS AgentCore

AI – Artificial Intelligence

For a Voice Agent, AI really means a few different things: speech recognition, text generation, text-to-speech, routing logic, and more. The LLM is only one part of the system.

ASR / STT – Automatic Speech Recognition and Speech-to-Text

These are often used interchangeably. It is the part that turns the caller’s audio into text that the LLM can understand.

In this project, the pipeline uses Cartesia STT (Ink model). Twilio ConversationRelay can also do ASR for you if you use that product instead of raw Media Streams. Other common vendors are Deepgram, Google Cloud STT, AssemblyAI, and OpenAI Whisper-style models.

The practical thing to remember: STT quality is not only about model accuracy. End-of-turn timing, background noise, phone audio, accents, and whether the user interrupts all matter.

Barge-in

Barge-in is when the caller interrupts the agent while the agent is speaking.

This is normal human behaviour. The system has to decide whether to stop speaking, listen, or update the conversation state.

CallSid / StreamSid

These are Twilio identifiers.

CallSid identifies the call. StreamSid identifies the media stream.

Relevant docs: Twilio CallSid and StreamSid

Cartesia

Cartesia is the speech and voice agent platform in this project.

In this project, Cartesia provides the speech components: STT for converting caller audio into text and TTS for converting the model’s response back into audio.

Cartesia can also handle more of the voice application layer: call orchestration, interruptions, deployment, logs, and call analytics (see Cartesia Line).

Cartesia Line

Cartesia Line is a code-first voice agent platform.

Cartesia Line can handle voice/call orchestration, Ink STT, Sonic TTS, interruptions, deployment, logs, and call analytics.

Line also has useful built-in concepts like transfer_call, send_dtmf, and handoffs (if you manage your telephony through them).

Note: If you are using your own Twilio number (which you can easily import with/connect to the Cartesia platform) and stream path, you still need access to the live Twilio REST API to manage the call controls.

Relevant docs: Cartesia Line

Conference

A conference is a Twilio call room with participants.

For warm transfer, the caller can be parked in a conference with hold music or a wait message while the human leg is being dialled. When the human agent accepts the transfer, they join the same conference.

The details matter:  startConferenceOnEnter,  endConferenceOnExit,  waitUrl, status callbacks, and what happens if a participant leaves.

Relevant docs: TwiML <Dial><Conference>.

ConversationRelay

Twilio ConversationRelay is a higher-level Twilio voice AI product (compared to Media Streams).

Instead of sending raw phone audio to your app, Twilio can handle STT and TTS and send structured WebSocket (text) messages to your application. Your app receives prompts/transcripts and sends text tokens back.

That can simplify the audio side. The tradeoff is that you are using Twilio’s ConversationRelay workflow rather than building your own Pipecat audio pipeline.

Relevant docs: Twilio ConversationRelay

DTMF – Dual-Tone Multi-Frequency.

This is the phone keypad signal: 0-9, star, and hash.

In this project, DTMF matters for warm transfer. The human agent answers, hears a brief, and presses 1 to accept the transfer. That is much safer than just dumping the caller onto whatever number was dialled.

Twilio <Gather> can collect DTMF digits.
Cartesia Line has a send_dtmf tool for pressing buttons when the voice agent itself needs to navigate a phone menu.

Relevant docs: Twilio <Gather>  and Cartesia Line  send_dtmf 

E.164

E.164 is the international phone number format, like +614….

It matters because APIs expect phone numbers to be normalised. Transfer tools should store and compare numbers in a predictable format.

HMAC

Hash-based Message Authentication Code is a cryptographic method used to simultaneously verify a message’s integrity and authenticity.

HMAC is used in the current app for the stream token. The /voice webhook validates Twilio’s signature, creates a call state, then mints a short-lived token tied to the CallSid. Twilio echoes that token in the WebSocket start.customParameters, and /ws validates it.

Why not just rely on Twilio’s webhook signature? Because the WebSocket upgrade does not carry the same signed form payload as the /voice POST.

IVR – Interactive Voice Response.

This is the classic “press 1 for…” phone menu. A voice AI receptionist can replace some IVR flows.

In our case, the human agent acceptance step is basically a tiny IVR: “press 1 to accept”.

Latency

Latency is a delay.

In a voice bot, latency is made up of many small delays:

• Twilio media to app.
• STT.
• VAD/end-of-turn.
• LLM first token.
• Tool calls, like Notion lookup.
• TTS first audio.
• Audio buffering back to Twilio.

The Notion (or other data store) lookup work is a good example. A live scan across Notion databases during the greeting made the call feel slow. The better approach was a quick caller lookup first, then background refresh/enrichment, with caching for slower data.

LLM – Large Language Model.

This is the reasoning layer. It reads the conversation, decides what to say, and calls tools when needed. In this project, the model has been Gemini.

The model should not be allowed to invent call control. It needs tools with clear rules: identify the caller, find the routing recipient, create a message, list assignments, cancel an assignment, and transfer a call.

Mark / Clear

mark is how the app asks Twilio to tell it when previously sent audio has finished playing. clear clears buffered audio.

These are Twilio Media Streams messages.

For transfer, mark was the safer way to avoid cutting off the “please hold” sentence when a caller is being transferred. Without it, the app can redirect the call before the caller hears “I will transfer you now”.

Relevant docs: Twilio mark and clear messages

Media Streams

Twilio Media Streams lets you stream call audio to your WebSocket server.

There are unidirectional streams and bidirectional streams. For the voice bot, bidirectional is the useful one because the app needs to receive the caller audio and send the agent audio back.

Twilio sends JSON messages like connected, start, media, mark, and stop. The audio payload is base64. When you send audio back to Twilio, you send media messages. When you want to know that buffered audio has finished playing, you send a mark and wait for Twilio to echo it.

Relevant docs: Twilio Media Streams

Notion

Notion is used as the operational data source in this project.

The bot looks up callers, contractors, clients, assignments, HR directory entries, routing recipients, and message logs from Notion.

Observability

Observability is how you understand what happened during a call.

For voice AI, logs alone are not enough. You want spans and metrics around:

• inbound webhook.
• WebSocket start/stop.
• STT latency.
• VAD/end-of-turn timing.
• LLM latency and tool calls.
• TTS latency.
• transfer lifecycle.
• call cleanup.

Relevant docs: Pipecat metrics

OAuth / JWT

OAuth and JWT are common auth patterns.

They matter in voice AI because every real-time endpoint is exposed to the internet. A public wss:// endpoint that accepts audio should not accept random traffic just because it looks like Twilio.

In this project, Twilio webhook signatures plus HMAC stream tokens were enough for the current Media Streams path. If using another runtime, hosted agent, or browser client, OAuth/JWT-style auth might be more relevant.

PCMU / mu-law / 8 kHz

PCMU, also called G.711 mu-law, is a common phone audio codec.

Twilio Media Streams sends and receives base64-encoded audio/x-mulaw at 8000 Hz for media frames. If you send the wrong audio format back to Twilio, the caller does not get useful audio.

This was one reason a direct “Twilio to some random WebSocket agent runtime” idea was not enough. The server must speak Twilio’s media message protocol and handle the expected audio format.

Pipecat

Pipecat is a framework for building real-time voice and multimodal agents.

In this build it provides the pipeline:

transport input -> STT -> user aggregator / VAD -> LLM -> TTS -> transport output -> assistant aggregator

The useful thing about Pipecat is that it gives names and structure to the pieces. The dangerous thing is assuming the transport protocol is magic. Twilio Media Streams still has its own JSON messages, audio format, CallSid, StreamSid, marks, and stop events.

Relevant docs: Pipecat

PII

Personally Identifiable Information.

Voice AI systems touch a lot of it: names, phone numbers, call recordings, transcripts, assignments, clients, and messages.

In this receptionist project, the agent should not volunteer surnames, phone numbers, or private details. It can use retrieved data to route the call, but it should speak only the minimum needed. For example, first names are usually enough for a transfer offer.

This is not just a prompt issue. Logs, traces, tool results, and evaluation datasets need the same discipline.

PSTN

Public Switched Telephone Network.

This is the normal phone network. When a caller dials a Twilio number from a mobile or landline, they are coming from the PSTN.

The annoying bit is that the PSTN is narrowband and phone-oriented. You are usually dealing with 8 kHz phone audio somewhere in the path, not clean studio audio.

Relevant vendors: Twilio, Telnyx, Vonage

RAG

Retrieval-Augmented Generation.

RAG means retrieving relevant knowledge and giving it to the model before it answers. In this project, Notion lookup is not exactly a generic vector RAG flow, but the idea is similar: fetch caller, contractor, client, assignment, HR directory, and routing context so the model can make the right decision.

The important bit is privacy. Retrieved data should be scoped. The agent should not volunteer surnames, phone numbers, or private records. If a contractor calls, they can get information relevant to their own assignments, not everyone else’s.

RTP

Real-time Transport Protocol.

RTP is how media packets often move in VoIP systems. You may not touch it directly if you are using Twilio Media Streams or ConversationRelay, but it is underneath a lot of telephony.

When people talk about codecs, jitter, packet loss, and media paths, RTP is usually close by.

SigV4

AWS Signature Version 4.

This came up when looking at AWS AgentCore. Direct Twilio-to-AgentCore looked tempting, but it did not fit.

The problems were:

• Twilio <Stream url> cannot practically sign the WebSocket request with AWS SigV4 headers.
• Twilio <Parameter> values arrive inside the later start.customParameters message, not in the initial WebSocket handshake where AgentCore auth would need them.
• Even if auth worked, the protocols did not match. Twilio sends Twilio Media Streams JSON with phone audio. Agent runtimes usually expect their own WebSocket protocol.

The conclusion was simple: you need a bridge or a single service that speaks Twilio on one side and your agent runtime on the other.

Relevant vendors/frameworks: AWS AgentCore, Twilio Media Streams, custom relay service.

SIP

Session Initiation Protocol.

SIP is a signalling protocol used to set up and control voice calls. If you are connecting phone systems, PBXs, trunks, and carriers, SIP eventually appears.

For this receptionist project, I mostly stayed at the Twilio Voice API/TwiML layer rather than managing SIP directly. That was enough for inbound calls, Media Streams, call redirects, and conferences.

State store

State is what the app remembers about the active call.

For a single container, in-memory state can work during local testing. In production, Redis or another shared store becomes important if there are multiple instances or restarts.

In this project state includes the caller, public base URL, allowed transfer numbers, active transfer state, stream token, and whether the call has already been redirected or ended.

Stop event

Twilio sends a stop event when the stream stops or the call ends.

This matters for cleanup. If the caller hangs up and the app does not cancel the pipeline, the model/STT/TTS work can keep running for no reason. Hangups need to cancel the active pipeline, clear state, and avoid trying to speak into a dead call.

Telnyx

Telnyx is another telephony provider.

We looked at Telnyx as an alternative to Twilio. The conclusion was similar for direct-to-AgentCore: Telnyx can stream audio and gives strong call control APIs, but stream_url and stream auth are still about connecting Telnyx to your server. They do not magically make a third-party agent runtime speak Telnyx’s protocol.

So Telnyx can be a good provider, but you still need a relay/bridge unless the voice agent framework directly supports its stream format.

Relevant docs: Telnyx Call Control

Tool calling

Tool calling is when the LLM asks the app to perform an action.

For this receptionist, tools are not optional decoration. They are how the agent does real work:

• identify the caller by phone number.
• find a routing recipient.
• list assignments.
• cancel an assignment.
• create a message.
• start a warm transfer.

The rules around tools matter. For example, transfer_call should only work for previously selected, call-capable recipients. The model should not be allowed to transfer to a random number it heard in the conversation.

Relevant frameworks: Gemini tool declarations, Pipecat tool handlers, Cartesia Line tools, OpenAI/Anthropic tool calling.

TTFB / TTFA

Time To First Byte and Time To First Audio.

For voice, TTFA is often the more useful mental model. The caller does not care when the server got a byte. They care when they hear something useful.

This is why streaming matters. Stream the LLM response if possible. Start TTS early if possible. Do not block the first greeting on slow CRM/database enrichment unless that data is needed to say hello.

TTS

Text-to-Speech.

This is the part that turns the agent’s text back into audio. In the current Pipecat setup, Cartesia TTS does this. Cartesia calls its TTS model family Sonic.

Twilio ConversationRelay can use TTS providers like Google, Amazon, Deepgram, and ElevenLabs depending on configuration and access.

The practical thing to remember: TTS is where latency becomes very obvious. A text chatbot can pause for a second and nobody really notices. A phone caller notices.

Turn-taking

Turn-taking is the question of when the caller’s turn ends and the agent’s turn starts.

Text chat mostly avoids this. Voice does not. A person can pause mid-sentence, say “um”, talk over the agent, press a key, or hang up while the model is thinking.

Good turn-taking needs VAD, STT partial/final transcripts, interruption handling, and prompts that tell the model what it has already said.

Twilio

Twilio is the telephony platform in this project.

It owns the public phone number, receives the inbound PSTN call, calls the /voice webhook, opens the Media Streams WebSocket, sends caller audio to the app, receives generated audio back, and performs the warm transfer call control.

The important thing I had to learn is that Twilio is not just a dumb audio pipe. Twilio has its own call state, identifiers, webhook signatures, TwiML instructions, media stream protocol, status callbacks, conferences, and REST APIs. If you want to move a live caller from an AI stream into a human warm transfer, you need to use Twilio call control deliberately.

Relevant products/frameworks: Twilio Programmable Voice, Twilio Media Streams, TwiML, Twilio REST API, Twilio Conferences, ConversationRelay.

TwiML

Twilio Markup Language.

TwiML is XML that tells Twilio what to do with a call. The webhook gets an inbound call, returns TwiML, and Twilio executes it.

Examples:

<Response> <Connect> <Stream url="wss://example.com/ws" /> </Connect> </Response>

or:

<Response> <Gather numDigits="1"> <Say>Press 1 to accept the call.</Say> </Gather> </Response>

The important bit: TwiML is not just a response format. It is call control.

Relevant docs: Twilio TwiML for Programmable Voice.

VAD

Voice Activity Detection.

VAD decides whether someone is speaking or not. That sounds small, but it is one of the things that makes a voice bot feel natural or painful.

If VAD ends the turn too early, the bot interrupts. If it waits too long, the call feels slow. If it treats background noise as speech, the agent can get confused.

In the Pipecat build, Silero VAD is used in the LLM user aggregator path. Cartesia Line handles turn-taking and interruption detection at the platform layer.

Relevant frameworks: Silero VAD, Pipecat VAD,

Warm transfer

Warm transfer means the caller is not blindly transferred.

The agent first gathers enough context, then contacts the human, gives them a short brief, and only connects the caller if the human accepts.

In this project, the warm transfer flow uses Twilio call control:

1. The model calls the transfer tool.
2. The app prepares transfer state.
3. The bot tells the caller to hold.
4. The app waits for a Twilio media mark confirming the hold prompt played.
5. The app redirects the live caller call into a conference.
6. The app calls the human agent.
7. The human hears a brief and presses 1.
8. Twilio joins the human to the conference.
9. If no answer, busy, failed, or declined, the caller returns to the AI to leave a message.

This is more work than a cold transfer, but it avoids a bad caller experience.

Webhook

A webhook is an HTTP callback from a service to your app.

Twilio uses webhooks heavily. Incoming call? Webhook. Human leg answered? Webhook. Gathered digit? Webhook. Transfer status changed? Webhook. Conference event? Webhook.

The important bit is that Twilio signs webhook requests using the public URL it called. If your app validates the wrong URL because it sees an internal Cloud Run URL or localhost, validation fails.

In our app, request validation builds candidate public URLs from PUBLIC_BASE_URL, PUBLIC_HOST, forwarded headers, host headers, and the ASGI URL.

Relevant docs: Twilio webhook security

WebSocket / WSS

WebSocket is the long-lived connection used for real-time media or event streams. WSS is WebSocket over TLS.

For Twilio voice AI, the URL must be public and secure. Localhost does not count unless it is behind a public tunnel. Cloud Run, ngrok, and load balancers can also make this confusing because the app may see an internal http://… URL while Twilio called the public https://… URL.

In the current app, /voice returns TwiML that points Twilio at /ws, and /ws validates a short-lived stream token before accepting the call media.

The post The ABCs of Voice AI Agents appeared first on ISbyR.