The problem
The client is a mid-size fintech processing consumer lending applications, with a support team of fourteen agents handling roughly 2,800 tickets a week. Around 60% of that volume was repetitive: application status questions, document requirements, payment schedule clarifications, and basic account access issues. That tier-one volume pushed genuinely complex cases (fraud disputes, hardship requests) into a backlog that regularly ran three to five business days.
They had tried a rules-based chatbot from a vendor eighteen months earlier. It handled maybe 15% of volume before agents started routing around it, because it couldn't reason over account-specific context. It could answer "what documents do I need?" but not "what documents do I still need for my application?", which was the question people actually asked.
The constraint that shaped everything else: this is a regulated lender. Every automated response touching account or application data needed to be traceable to a specific source, reviewable after the fact, and incapable of fabricating information about a person's application status. A hallucinated answer about loan terms isn't a minor UX problem here; it's a compliance and legal one.
Why a generic chatbot wasn't the answer
Before committing to an architecture, we spent the first two weeks of discovery mapping the actual ticket distribution against what each category required to resolve correctly. Three findings shaped the design:
- Most tickets needed account-specific retrieval, not general knowledge. "What's my application status" requires querying the client's core loan origination system in real time, not answering from a static FAQ.
- The knowledge base was fragmented and partly outdated. Policy knowledge lived in three places (a wiki, a shared drive of PDFs, and tribal knowledge in a Slack channel), and roughly 20% of what agents cited from memory wasn't written down anywhere.
- Every response needed an audit trail. Compliance required knowing exactly what source material informed any automated answer touching application or account data, retrievable on demand for up to seven years.
This ruled out a purely fine-tuned model (no live account data access, and fine-tuning would bake in a point-in-time snapshot of policy that goes stale). It also ruled out a simple wrapper around a general-purpose chat API (no retrieval grounding, no audit trail, and no way to stop it answering outside its actual knowledge).
Architecture
We built a retrieval-augmented agent with a strict separation between "what the model is allowed to say" and "what data it can see," enforced at the retrieval layer rather than trusted to prompt instructions alone.
Retrieval layer
Two separate retrieval paths feed the agent, kept deliberately distinct:
- Policy and documentation retrieval. The fragmented knowledge base was consolidated into a single source of truth, chunked by policy section (not by document, since documents mixed unrelated topics), embedded, and indexed in a managed vector store. Every chunk carries metadata (source document, section, last-reviewed date, and internal approval status), so unapproved draft policy can never surface in a customer-facing answer.
- Live account data retrieval. A narrow, read-only API layer sits in front of the loan origination system, exposing exactly the fields a support answer needs (application status, required documents, payment schedule) and nothing else. This was the most time-consuming piece of the build: getting the client's platform team to expose a safe, rate-limited, read-only interface took longer than the AI work.
Orchestration
An intent classification step runs first, sorting each incoming message into one of five categories (status inquiry, document question, payment question, access issue, or escalate to human) with a small, fast model rather than the model that generates full responses. This keeps latency down on common paths and gives a clean point to force escalation for anything out of scope. Anything touching fraud, disputes, or account changes is hard-routed to a human agent, whatever the model thinks it can answer.
For the categories the agent does handle, it retrieves relevant policy chunks and, where needed, live account fields, then generates a response constrained to cite only retrieved content. We don't allow open-ended generation about account specifics: the response template requires the model to cite which retrieved fact supports each claim, and that is logged alongside the response.
Guardrails and audit
Every automated response is logged with its full retrieval context: which policy chunks and account fields were used, the classified intent, and a confidence score from the retrieval step. When confidence is below threshold, or retrieval finds no matching policy chunk, the ticket goes to a human agent instead of getting a best-effort answer. That's the mechanism that stops the system from answering questions it has no grounded information for.
Challenges and tradeoffs
- Chunking policy documents correctly took three iterations. Our first pass chunked by fixed token count, which often split a policy rule from its exception clause, and the model would confidently answer with just the rule. Structure-aware chunking that respects section boundaries fixed most of it, but we still had to manually re-review about 15% of the knowledge base, where the original documents had no clear section structure.
- Live API latency was the biggest performance risk. The loan origination system's API had unpredictable response times under load, sometimes over 2 seconds. We added a short-lived cache for account status lookups (90-second TTL) to keep the common case fast, and bypassed it for data where freshness matters, like a payment status change.
- The confidence threshold needed real tuning, not a guess. We initially set it conservatively high, which sent nearly 40% of eligible tickets to humans unnecessarily. We spent a week in the prototype phase having a human reviewer score a sample of automated responses to find where the confidence score actually correlated with correctness, and lowered the threshold once that was established.
Results
The system deflects 61% of tier-one ticket volume without a human touching it, measured over the first ten weeks post-launch. The escalation logic works as intended: across the tickets it handled, a post-launch audit sample found no case of the agent giving incorrect information about a customer's application or account, because anything it wasn't confident about went to a human instead of being guessed.
The backlog for genuinely complex cases, the ones that need a person, dropped from three to five business days to under one, because agents no longer spend most of their day on repetitive status questions. The client's compliance team completed their first quarterly audit of the system's logs without requesting any changes to the logging or escalation design, which was treated internally as validation that the audit trail approach was sufficient.
What we'd do differently
In hindsight, we'd push to start the platform team's read-only API work in parallel with discovery rather than after architecture sign-off. It was on the critical path and became the real bottleneck for the first month: the AI and retrieval work was ready weeks before there was live data to integrate against.