The problem
The client runs a digital media and publishing platform with AI-assisted editorial tools built up over roughly eighteen months: headline suggestion, article summarization for a mobile digest product, tag and category classification, and a few other tasks. All of them called the same large frontier model regardless of task complexity, mostly because it was the model the team started with and nobody had revisited the decision as the product grew. The monthly inference bill had grown in step with usage, and a cost review flagged that a meaningful share of spend went to tasks, like category classification, that didn't obviously need a large model.
Simply swapping everything to a cheaper model would have been the wrong move, and the client's engineering lead knew it. Some tasks, like nuanced headline suggestion for sensitive news, genuinely benefited from a larger model's judgment, and a blanket downgrade risked degrading exactly the tasks where quality mattered most to editors. The problem needed a routing decision per task category, backed by real quality comparison, not a single global model swap.
What we actually did
We built a routing layer between the client's editorial tools and the model providers. It classifies each incoming request by task type and complexity signals, then routes it to the smallest model shown to perform acceptably for that task category.
Weeks 1-2: benchmarking every task category against three model tiers
Before changing any routing, we ran the existing large model, a mid-tier model, and a small, fast model on a sample of real historical requests across all six editorial task categories. The client's editors blind-scored the outputs without knowing which model produced which. The finding was genuinely useful: category and tag classification was statistically indistinguishable across all three tiers, while nuanced headline suggestion for sensitive or breaking news showed a real quality drop on the smallest model that editors noticed.
Weeks 3-4: building the router and a fallback path
The router classifies each request by task type (a reliable signal, since each editorial tool calls a specific endpoint) and, for task types where complexity varies, a lightweight secondary signal like input length or a "sensitive topic" tag set by the editorial team. We also built an automatic fallback to the next tier up when the smaller model's confidence signal is low, so an unusually complex request in an otherwise routine category still gets escalated.
Week 5: staged rollout with a quality-monitoring dashboard
We rolled routing out one task category at a time rather than switching all six at once, starting where the benchmarking showed the clearest case for a smaller model (classification and tagging), and kept headline suggestion on the original large model. Each category's rollout included a live dashboard tracking editor override rates (how often an editor rejected or heavily edited the AI's suggestion) as an ongoing proxy for quality, rather than relying only on the initial benchmark.
Challenges and tradeoffs
- Blind scoring by editorial staff took real coordination and pulled people off their regular workload. Unbiased comparisons meant editors scoring outputs without knowing which model produced them, which required a lightweight blind-review tool and real editor time away from daily production. The client's managing editor sponsored this internally, and that mattered more for honest participation than anything in the tooling.
- The confidence-based fallback added latency variability that needed its own monitoring. Requests that escalated from a small to a larger model took measurably longer than those handled by one tier, which showed up as inconsistent response times in early testing. We added a latency dashboard split by whether a request escalated, so the team could tell routing overhead apart from the router doing its job.
- Holding one entire task category out of routing entirely was a harder sell internally than it should have been. Cost savings drove the whole project, and leaving the highest-volume category on the most expensive model looked, on a spreadsheet, like leaving money on the table. We walked the client's finance stakeholders through the benchmarking data rather than just asserting the recommendation. It won buy-in but took an extra review cycle we hadn't scoped.
Results
Overall inference spend dropped 47% in the two months after full rollout across the five routed task categories, with category and tag classification, the highest-volume of the routed categories, producing the largest absolute savings. Editor override rates in the routed categories moved by less than 0.3 percentage points from their pre-routing baseline, within what the editorial team considers normal variation rather than a quality regression.
Headline suggestion, held out of routing based on the benchmarking, still runs on the original larger model. After seeing the full cost-versus-quality picture, the client's finance team told us they were glad the recommendation hadn't been overridden for a bigger number on the savings report.
What we'd do differently
We'd build the latency dashboard alongside the router in weeks 3-4 rather than adding it once inconsistent response times showed up in testing. It didn't cause a production problem, but we spent a few days retroactively instrumenting something that could have shipped with the router's first version.