The problem
The client operates a regional restaurant chain where call-in ordering is still a meaningful share of volume, especially at peak hours when in-store staff are stretched thin and phone orders either go to voicemail or get handled quickly and inconsistently by whoever picks up. Order accuracy varied by location and by who answered. Upselling (suggesting a side, a larger size, a current promotion) happened only when a staff member remembered and had time, which meant it rarely happened during the busiest windows, exactly when it would have mattered most.
The client wasn't looking to replace phone ordering with a kiosk-style app; its customer research showed a real segment of callers preferred talking to someone over navigating an app, particularly when customizing an order. The ask was a voice agent that takes an order as naturally as a good staff member would, including the upsell a good staff member remembers to make, without customers feeling sold to by a script.
What we actually did
We built a streaming voice agent that handles the full order flow (greeting, order taking, customization, upsell, and confirmation) with a low-latency speech pipeline and, more important for the business outcome, upsell logic based on what the current order and time of day make relevant rather than a fixed script.
Weeks 1-2: streaming pipeline and barge-in as the technical foundation
We built the speech-to-text, LLM, and text-to-speech pipeline as a streaming system from the start, with proper barge-in support, so a customer interrupting mid-response (to correct an item, for instance) feels natural instead of having to wait for the agent to finish. Median end-to-end response latency landed at 1.1 seconds; in internal testing, that was roughly the point below which callers stopped reacting to pauses and simply ordered naturally.
Weeks 3-4: context-aware upselling instead of a fixed script
Rather than a fixed "would you like fries with that?" on every order, the upsell logic draws on the specific order, the time of day, and what that location is promoting that week. A customer ordering a sandwich without a drink gets a drink suggestion; a customer ordering during a promotional window hears that promotion once, naturally, and not again if they decline. The client's ops team wanted the agent to try again after a declined upsell, but on our recommendation upsells are capped at one per order.
Weeks 5-7: order-system integration and phased multi-location rollout
The agent integrates directly with the client's point-of-sale system, so a completed voice order reaches the kitchen exactly like a staff-taken order, with no separate reconciliation step. We rolled out to three pilot locations first, comparing order accuracy and average order value against those locations' historical phone orders, before expanding to the full chain. A chain-wide rollout of an untested upsell approach carried real brand risk if the tone came across as pushy at scale.
Challenges and tradeoffs
- Order customization language varies a lot by how real customers actually talk, not how a menu is written. Customers describe modifications in ways that don't match menu terminology ("no crunchy stuff," meaning no fried onions, was one real example from pilot testing), and early testing showed a non-trivial misorder rate on unusual phrasing. We built a clarifying-question fallback for low-confidence customizations instead of guessing. It adds a few seconds to those calls but keeps wrong orders from reaching the kitchen.
- Background noise at drive-through-adjacent phone lines degraded speech recognition accuracy more than test-environment results predicted. Some locations' phone lines pick up real restaurant noise, and our accuracy numbers from quieter test conditions didn't fully hold at the noisiest pilot location. We tuned the speech recognition's noise handling with audio samples from that location rather than relying on a generic noise-robustness improvement.
- Kitchen staff needed to trust that voice-agent orders arriving in the POS were accurate, which took visible track record, not just integration testing. Early in the pilot, kitchen staff at one location double-checked every voice-agent order by hand, which slowed things down rather than speeding them up. The client's ops manager addressed this by sharing weekly accuracy numbers with kitchen staff directly, which built trust faster than anything we could have done on the engineering side.
Results
Average order value on agent-handled calls rose 31% compared with the same locations' historical average for human-staffed phone orders, measured over the first two months of the three-location pilot and holding steady through the first month of full rollout. 68% of calls were handled entirely by the agent; the rest transferred to a staff member at the customer's request or after a clarification the agent couldn't resolve with one follow-up question.
The one-upsell-per-order cap, the main point of disagreement during the build, is now cited by the client's marketing team as central to why the feature hasn't drawn the customer complaints they'd braced for: the upsell reads as a suggestion, not a sales tactic.
What we'd do differently
We'd collect real ambient-noise audio from each pilot location before tuning speech recognition, rather than tuning against generic noise profiles and correcting once a location's real conditions showed a gap. The fix was quick once we had the right audio, but location-specific samples should have been on the original pilot-prep checklist.