The problem
The client operates a multi-location physical therapy practice whose front desk answered roughly 400 calls a day, a large share of them scheduling, rescheduling, and basic intake questions ("do you take my insurance?", "what should I bring to my first visit?"). Phone volume regularly overwhelmed the two-person front desk at peak hours, sending a meaningful share of callers to voicemail, and follow-up was slow enough that some callers booked with a competitor instead.
The client had looked at a basic IVR ("press 1 for scheduling") and rightly rejected it. A phone tree can't handle a caller saying "I need to move my Thursday appointment because something came up," and forcing patients through rigid menus for a two-minute call was a worse experience than the problem it solved.
Why latency was the actual design constraint, not accuracy
Voice is a different problem from chat. A text agent can take a second or two to respond and feel normal; a voice agent with the same delay feels broken, because silence on a phone call reads as a dropped connection, not thinking time. We scoped the engagement around latency from day one, because a voice agent that's accurate but slow fails no matter how good its reasoning is.
Architecture
Streaming pipeline, not request-response
The system uses a streaming speech-to-text, LLM, and text-to-speech pipeline in which each stage starts processing before the previous one finishes, instead of waiting for a complete transcription before generating a response. This is the single biggest latency lever available, and it's why the model's time to first token is 340 ms instead of the multi-second delay a naive request-response chain would produce.
Real-time calendar integration
The agent has live, scoped read-write access to the practice's scheduling system through an API integration, so it can check actual therapist availability and book, reschedule, or cancel appointments during the call instead of collecting a request for a human to process later. That's the difference between a genuinely useful voice agent and an expensive answering machine.
Confident handoff to a human, not a dead end
Detailed insurance questions (beyond whether an insurer is accepted), clinical questions, and anything where the agent's confidence drops mid-call trigger an immediate handoff to the front desk, with a summary of the call so far so the caller doesn't repeat themselves. We treated graceful handoff as a core feature, not a fallback bolted on afterward, because a caller stuck with a confused voice agent and no way out is worse off than one who never met it.
Call recording, transcription, and QA loop
Every call is recorded and transcribed, feeding a weekly review in which the practice's office manager listens to a sample of successful and escalated calls. It has been the main way edge cases get caught and fixed after launch (an insurance provider name the system kept mishearing, a scheduling rule it didn't know about).
Challenges and tradeoffs
- Background noise and phone audio quality genuinely degraded transcription accuracy. Calls from a patient's car or a noisy environment produced meaningfully worse transcription than clean audio, so we tuned the confidence threshold to route unclear audio to a human rather than guess at a garbled scheduling request.
- Barge-in (a caller interrupting the agent mid-sentence) needed explicit handling. Natural phone conversation involves interruption, and an agent that can't be interrupted feels robotic and frustrates callers. Supporting barge-in properly, so the agent stops talking and actually listens, added real complexity to the streaming pipeline.
- Insurance provider name recognition needed a custom vocabulary. Generic speech-to-text struggled with some regional and less common insurance provider names. We added a custom vocabulary of the practice's accepted insurers, which measurably improved recognition on this high-frequency phrase category.
Results
Over the first two months after rollout, 73% of calls were handled entirely by the voice agent with no human involvement, covering the large share of calls that are routine scheduling and basic intake. Average end-to-end response latency of 1.4 seconds was comfortably within what callers in usability sessions described as a normal conversation rather than a delayed one.
The front desk team, initially wary that the system was meant to replace them, now spends most of its time on complex calls and in-person patients. The office manager credits this with the improvement in team morale once staff saw it taking repetitive volume off their plate, not their jobs.
What we'd do differently
We'd build the custom insurance-provider vocabulary list during initial development rather than after launch. It was a foreseeable, high-frequency phrase category specific to this practice, and a more thorough initial data-gathering pass would have avoided a week of post-launch accuracy tuning.