The problem
The client runs a project management SaaS product with several years of accumulated features, integrations, and settings, so many that their support team's most common ticket category was some version of "how do I find the setting for X?" or "where is the feature that does Y?" Their existing search was a straightforward keyword match against titles and tags, which worked for exact matches but failed on anything phrased differently from the feature's internal label.
The ask was AI-powered natural-language search: let a user type "how do I stop getting notified every time someone comments" and land on the actual notification settings page, even if "stop" or "comments" doesn't appear in that page's title or tags. As with most integration work in an established product, it had to work inside the existing product, on the existing release cadence, without a search infrastructure rewrite.
Why we scoped this as retrieval over structured content, not a chatbot
The client's initial framing was "add an AI chatbot for help." We pushed back on that framing. A chatbot introduces a new interaction pattern users have to learn, while the underlying need was better search, which fits a pattern (the search bar) users already knew. We scoped it as upgrading the existing search with semantic retrieval rather than adding a new surface: a better user experience and a smaller integration footprint.
Architecture
Compatibility audit before writing code
We spent the first week entirely on the client's existing backend: how their content (feature pages, settings, help docs) was structured and stored, what the search infrastructure did, and where a new retrieval layer could plug in without disrupting the indexing pipeline their engineers maintained. The audit revealed that help documentation and in-app settings pages lived in two separate systems with no shared content model, which directly shaped the integration design.
Unified retrieval index across two content sources
We built an embedding-based retrieval index covering both the in-app settings and feature pages and the separate help-documentation system, normalized into a common schema so a single query surfaces results from either source ranked together, rather than two separate searches or two disconnected result lists.
API-first integration into the existing backend
The new semantic search runs as an additional API endpoint alongside the existing keyword search, not a replacement requiring a cutover. The search bar's frontend was updated to call the new endpoint, with keyword search kept live as an automatic fallback if the semantic service is unavailable, a deliberate reliability decision for a live, revenue-generating product.
Feature-flagged, zero-downtime rollout
The new search shipped behind a feature flag and rolled out gradually (10%, then 50%, then 100% of users) on the client's normal weekly release cadence, with the ability to revert instantly to keyword-only search if anything looked wrong. That's what let it ship with zero downtime and no dedicated maintenance window, folded into releases that were already happening.
Challenges and tradeoffs
- The two-content-source normalization was more work than the search feature itself. Getting settings pages and help documentation into a genuinely comparable schema, with consistent metadata for ranking, took nearly half the total engineering time. That's common in integration work: the AI feature is often the smaller piece next to the data plumbing needed to feed it.
- Ranking results across two very different content types needed real tuning. A help article and an in-app settings page have different natural lengths and structures, and naive embedding similarity initially over-favored longer help articles regardless of relevance. We added content-type-aware ranking adjustments after this surfaced in internal testing.
- The feature flag rollout caught a real edge case at the 50% stage. Some enterprise customers' custom-configured settings pages weren't indexed correctly because of a data model difference we hadn't accounted for in the initial audit. Catching it at 50% rather than 100% limited the impact and let us fix it before full rollout.
Results
The feature went from scoping to 100% availability in five weeks, entirely within the client's release cadence, with zero downtime across the staged rollout. Support tickets in the "how do I find X?" category dropped 31% in the two months after full rollout, which the support lead called the clearest before-and-after signal that the feature worked, beyond any internal usage metric.
Because it was integrated as an additional endpoint, with keyword search preserved as a fallback, the client's search analytics and the rest of the product needed no changes, which kept engineering risk and QA scope contained.
What we'd do differently
We'd extend the compatibility audit in week one to explicitly include enterprise-tier custom configurations, not just the standard product schema. The custom-settings edge case at the 50% stage came from a gap in how thoroughly we'd audited configuration variance across customer tiers, and a more complete audit would have caught it before rollout rather than during it.