The problem
The client is a well-funded AI-native startup whose product ships several LLM-powered features. Its monthly model API bill had grown fourfold over two quarters without anything like matching growth in users, which is why they brought us in. Nobody on the small engineering team could see where the spend was going (which features, model tiers, or code paths were driving it), because there was no centralized cost monitoring, just a single combined monthly invoice.
There was a reliability gap too. When a model provider had a brief outage or degraded performance, the team found out from user complaints rather than their own monitoring, because nothing tracked latency or error rates for the AI-dependent features separately from general application monitoring.
Why we started with an audit, not a rebuild
The instinct with a cost problem is often to jump to optimization: smaller models, aggressive caching, rate limiting. We insisted on a two-week audit first, instrumenting cost and usage tracking across every AI-dependent code path before changing anything. Optimizing without knowing where the cost is concentrated risks solving the wrong problem well.
Architecture
Cost attribution by feature and code path
We added structured logging at every model API call in the codebase, tagging each call with the feature, user tier, and code path responsible, and fed it into a cost dashboard broken down by those dimensions. This was the single highest-value piece of the engagement: it took about a week to instrument and immediately exposed three cost leaks that had been invisible in the combined invoice.
What the audit found
- A retry loop with no backoff or cap. A background job that occasionally failed silently was retrying against the model API with no exponential backoff or retry cap. Under rare failure conditions it burned through calls for hours before anyone noticed, and this alone accounted for a meaningful share of anomalous spend in the prior quarter.
- An oversized model used for a simple classification step. A low-stakes internal routing decision (deciding which of three response templates to use) was running on the same frontier model as the product's main conversational feature, when a much smaller, cheaper model performed identically on this narrow task.
- No caching on a frequently-repeated query pattern. A significant share of incoming requests were near-duplicates of recent ones (users re-asking a slightly rephrased version of the same question), and none were cached, so every one hit the model API fresh.
Infrastructure-as-code for reproducible environments
Beyond the immediate cost fixes, we moved the client's model configuration (model versions, rate limits, fallback behavior per feature) into version-controlled infrastructure-as-code, replacing manual changes made directly in a provider dashboard with no change history or easy rollback.
Latency and drift monitoring with alerting
We built dedicated monitoring for AI-specific reliability signals (latency percentiles per feature, error rates by provider, and a basic drift check comparing recent output characteristics against a baseline), with alerts routed to the team's existing on-call system. A model provider issue now surfaces as an alert within minutes rather than a support ticket hours later.
Challenges and tradeoffs
- Instrumenting cost attribution touched more of the codebase than expected. Model API calls were scattered across the codebase without a consistent wrapper, so adding attribution tagging meant touching code in many places. We used the opportunity to introduce a thin shared client wrapper for all model calls, which makes future instrumentation and provider changes much simpler.
- The retry-loop bug was in a rarely-triggered failure path, which is exactly why it went unnoticed. It only activated under an intermittent upstream failure condition, which made it hard to catch in normal testing. The cost-attribution instrumentation had to run for about ten days before enough data accumulated to spot the anomaly clearly.
- Caching introduced a staleness tradeoff the team had to explicitly accept. Caching responses to near-duplicate queries meant occasionally serving a slightly stale answer to a rephrased question. We kept the cache TTL conservative and excluded any feature involving time-sensitive or personalized data from caching entirely, a deliberate tradeoff the team signed off on.
Results
AI infrastructure spend dropped 38% within six weeks of the audit starting, mainly from fixing the retry-loop bug and right-sizing the classification model, with caching adding a smaller but meaningful reduction. Detection of AI-feature incidents became 4.6 times faster, comparing time to first alert before and after the new monitoring (based on two incidents in each period, so a small sample).
The cost-attribution dashboard has become a standing tool the team checks weekly, not a one-time audit artifact. It caught a new anomaly on its own within a month of the engagement ending, which the client's engineering lead cited as proof that the instrumentation was the most durable part of the work.
What we'd do differently
We'd push for the shared model-call wrapper as a first step rather than a byproduct of instrumentation work. Introducing it earlier would have made cost-attribution tagging faster to add and surfaced the retry-loop bug's severity sooner, because every call would have been visible in one consistent place from day one rather than scattered across ad hoc call sites.