The problem
The client runs a mid-market B2B workflow platform with enough configuration surface (permissions, integrations, custom fields, notification rules) that new accounts genuinely need help setting up, but not enough to justify a full onboarding-services team at their price point. Their support data showed accounts in their first fourteen days generated close to a third of total ticket volume. When we sampled two months of those tickets, the overwhelming majority were a small set of repeat questions: how to invite teammates with the right permission level, how to connect existing tools, and how to migrate data from a spreadsheet or legacy tool.
Those questions had good answers already, scattered across help docs, a getting-started video, and institutional knowledge inside the support team's heads. The problem wasn't a lack of answers. A new user hitting friction at 11pm on a Tuesday wasn't going to dig through help docs; they'd open a ticket and wait until morning, and a meaningful fraction of them churned during that wait.
What we actually did
We built an in-product copilot that sits inside the setup flow itself, not a separate chat widget bolted onto the dashboard, because the friction was contextual. A user stuck on the permissions screen needs an answer about permissions right there, not a general-purpose assistant they have to explain their problem to from scratch.
Weeks 1-2: scoping what the copilot would and wouldn't touch
The first real decision was what the copilot could do versus what it could only explain. We scoped it to read the account's current configuration and explain next steps, but not to change configuration on the user's behalf, even though a "just do it for me" version was buildable and the client's product team initially wanted it. A permission change misapplied autonomously by an AI copilot in a paying customer's first week was an unacceptable failure mode compared with the support-ticket savings at stake.
Weeks 3-4: grounding answers in the account's actual state
The copilot's answers are grounded in two sources: a curated knowledge base built from the client's help docs (rewritten into a conversational, step-by-step format), and read-only access to the account's live configuration, so it can say "you haven't connected an integration yet" instead of a generic answer that ignores what the user has done. The second part took longer to build than the knowledge base, because it meant wiring the copilot into the internal APIs the dashboard uses, scoped so it could read state but never write it.
Weeks 5-6: handoff design and launch
We built an explicit, low-friction handoff to a human support rep for anything below a confidence threshold, shown as a single button rather than a failure state, so a user with a genuinely unusual problem isn't stuck arguing with a bot. Before launch, we ran the copilot against two weeks of real historical support tickets (reviewed by the support team, not live traffic) to see how often it would have deferred versus answered, and used that to calibrate the threshold before any real user saw it.
Challenges and tradeoffs
- The knowledge base needed a real rewrite, not a repurposing of existing docs. The client's help docs were written as reference material, not as answers to "what do I click next?" We spent more of weeks one and two than planned restructuring roughly sixty articles into a step-by-step format the copilot could reliably use, because the original docs produced technically correct but practically unhelpful answers.
- Read access into live account state raised a security review we hadn't scoped for at the original timeline. Because the copilot reads real configuration data, including some fields adjacent to billing and user permissions, the client's security team required a review of exactly which fields it could access before we connected it to production. That added roughly a week, which we absorbed without moving the launch date.
- Calibrating the confidence threshold for human handoff took real iteration, not a single number. Set too high, the copilot deferred to a human on questions it could easily answer, defeating the point. Set too low, it attempted genuinely ambiguous questions. We settled on a threshold only after three rounds of testing against real historical tickets, with the support team scoring each response.
Results
New-account support tickets dropped 42% in the eight weeks after launch, almost entirely in the permission-setup and integration-connection categories the copilot was built for. Tickets in other categories stayed roughly flat, which the support lead took as a sign that the drop was real rather than users giving up on filing tickets. Median time from account creation to a user's first completed core workflow became 3.2 times faster, because users resolved setup friction in the moment instead of waiting on a support queue.
The copilot made no account configuration change without an explicit user confirmation click. That was the metric the client's engineering leadership cared about most going in, because it was the failure mode they most feared shipping.
What we'd do differently
We'd loop the security team into the scoping conversation in week one rather than week four. The review itself wasn't the problem; it surfaced two reasonable, easy-to-accommodate field-access restrictions. But discovering them after the data-access design was mostly built meant rework that a fifteen-minute conversation in week one would have avoided.