An AI voice receptionist is a real-time distributed system wearing a friendly voice. Callen AI — our in-house product — answers live inbound calls for clinics, understands what the caller wants, checks a real calendar, and books the appointment, all before the caller loses patience. This post walks through the actual architecture: the telephony layer, the streaming speech pipeline, the LLM orchestration in the middle, and the production guardrails that make it trustworthy enough to put in front of someone else's patients.
We're publishing this because most "we build AI agents" pitches stop at a diagram with four boxes. The interesting engineering is in what happens between the boxes.
The core constraint: latency is the product
Everything in a voice agent's architecture is downstream of one number: the silence between the caller finishing a sentence and the agent starting its reply. Humans tolerate roughly a second of pause in phone conversation before it feels broken. That budget has to cover speech-to-text finalising the transcript, the LLM deciding what to say, text-to-speech generating audio, and every network hop in between.
You do not hit that budget with a naive pipeline that waits for each stage to finish. You hit it by streaming everything:
- Audio streams in continuously over a WebSocket, not in recorded chunks.
- Transcription is incremental — the pipeline sees partial transcripts while the caller is still talking.
- The LLM's reply streams out token by token.
- Speech synthesis starts on the first sentence of the reply, not the last — the caller hears the beginning of an answer whose end doesn't exist yet.
Once you commit to streaming at every stage, the architecture largely designs itself. Here is how the stages fit together.
Stage 1: Telephony ingress — Twilio as the front door
A clinic's number (existing or new) routes to Twilio. On an inbound call, Twilio opens a Media Stream — a WebSocket that carries the caller's raw audio to our backend in near real time, and accepts audio back in the other direction. That WebSocket is the spine of the whole call: everything else — transcription, reasoning, synthesis — hangs off it.
Using a telephony platform rather than raw SIP infrastructure is a deliberate boutique-team decision. Twilio gives us number provisioning, PSTN interconnect, call recording, and a clean failure model, and lets a small senior team spend its engineering budget on the part callers actually notice: the conversation.
Stage 2: Streaming speech-to-text
The inbound audio feeds a streaming STT engine that emits two kinds of events: partial transcripts (cheap, revisable guesses while the caller is mid-sentence) and final transcripts (committed text). Two details matter far more than raw word accuracy:
Endpointing — deciding that the caller has finished a thought. Cut in too early and the agent interrupts; wait too long and the pause feels dead. This is a tunable trade-off between snappiness and rudeness, and it needs adjusting on real call data, not lab audio.
Domain vocabulary. A generic model hears clinic-specific terms — treatment names, doctors' names — and produces creative nonsense. Biasing transcription toward each clinic's vocabulary is configuration work per tenant, and it is the difference between "book me for a cleaning" and a mis-heard mystery.
Stage 3: The brain — GPT-4 with tools, not a script
The middle of the pipeline is an LLM orchestration loop built around GPT-4-class models. Three design decisions define it:
Conversation policy lives in the system prompt; facts live in tools. The model is told who it is, what the clinic offers, and — just as important — what it must not do (diagnose, quote unlisted prices, promise a slot it hasn't verified). But it is never asked to remember the calendar. Anything factual is fetched at call time.
Tool calling instead of free generation. When the caller wants an appointment, the model doesn't answer from imagination — it emits a structured function call: check_availability(service, window), then book_appointment(...) against the clinic's actual booking system. The LLM decides when to call; deterministic TypeScript code decides what actually happens. This split — model proposes, code disposes — is the single most important pattern in production agent engineering.
Constrained conversational scope. Callen AI's job is reception: greet, understand, look up, book, take a message, hand off. It is explicitly not a medical chatbot. Narrowing the scope is what makes the long tail of a live phone call tractable — and it's why a vertical agent can be production-grade while a "do anything" bot stays a demo.
The orchestration layer around the model handles the unglamorous rest: per-tenant configuration (each clinic gets its own services, hours, staff, and voice), conversation state, retries on transient API failures, and timeouts so a slow dependency degrades the answer rather than freezing the call.
Stage 4: Text-to-speech — ElevenLabs, sentence by sentence
The reply streams from the LLM into ElevenLabs for synthesis, sentence by sentence, and the resulting audio streams straight back down the Twilio WebSocket. Natural prosody matters commercially, not just aesthetically: callers hang up on robotic voices, and a receptionist that sounds like a receptionist gets treated like one.
The subtle engineering here is barge-in. Real callers interrupt. The pipeline has to notice fresh inbound speech while the agent is talking, stop playback, discard the now-stale rest of the reply, and re-enter listening — the full stack flushed cleanly mid-utterance. Getting barge-in right is one of those problems that is invisible when it works and fatal when it doesn't.
The system of record: boring on purpose
Around the real-time pipeline sits a deliberately conventional SaaS backbone — TypeScript and Next.js, Supabase (Postgres) as the system of record, Stripe for billing, and self-serve onboarding that gets a clinic live in about 30 minutes. Every call produces a transcript, a structured outcome (booked / message taken / handed off), and a log the clinic can audit.
That boredom is a feature. The novel risk in the product is concentrated in the voice pipeline; spending novelty budget on the database would be architectural malpractice. We wrote more about that principle in AI is a multiplier, not a fix.
Guardrails: designing for the failure, not the demo
A production agent is defined by what happens when it's out of its depth:
- Human handoff as a first-class outcome. When the caller is upset, the request is out of scope, or confidence drops, the agent's job is a graceful exit: take a message with a callback promise, or forward the call. A handoff is a success mode, not a failure.
- Write-path validation. Booking mutations are validated by application code — real service, real slot, sane data — regardless of what the model proposed. The LLM cannot corrupt a calendar by hallucinating.
- Full observability. Recordings and transcripts feed a regression suite of real (anonymised) call scenarios, so a prompt or model change is tested against last month's hard calls before it meets a live caller.
- Data protection by scope. The agent collects the minimum needed to book — name, contact, service — under GDPR-conscious retention. Narrow scope helps here too: a receptionist that never gives medical advice also never invites clinical disclosures it shouldn't store.
What building our own product taught us
Callen AI is both a product and our proof of delivery. Running our own agent against real call volume taught us lessons no client engagement could have compressed:
- The demo is 20% of the work. A convincing voice-agent demo takes a good week. Production — barge-in, endpointing tuned on real audio, tenant isolation, billing, the regression suite — is where the other 80% lives.
- Latency work is architecture work. You cannot bolt streaming on later; the pipeline is either designed around it or condemned to awkward pauses.
- Vertical beats general. Every hard sub-problem gets easier when the agent's scope is one job done completely.
- Deterministic edges, probabilistic core. The LLM makes conversational decisions; typed, tested code owns every side effect. Callers get flexibility; the calendar gets correctness.
This is the same architecture we now build for clients — voice agents, RAG assistants, and multi-agent workflows — in Greek and English, integrated with the systems they already run. If you want an agent like this in front of your own callers, the fastest way to find out what it takes is a 20-minute scoping call.