The problem
The client is a regional insurance carrier processing property claims. An early triage step, classifying each incoming claim into one of eighteen categories (water damage, fire, theft, liability, and so on, each routed to a different specialized adjuster team), was done manually by a small intake team reading claim descriptions and supporting documents. At their volume, this created a one-to-two-day delay before a claim reached the right adjuster, which directly affected the customer satisfaction scores the client was measured on.
They'd tried a general-purpose model with a detailed prompt describing the eighteen categories, which reached roughly 76% accuracy. That wasn't enough to remove the manual review step, because misrouted claims created rework and customer frustration that outweighed the time saved. The categories were also domain-specific and overlapping in ways a general model struggled with, particularly several water-damage categories with different coverage implications but similar surface language.
Why fine-tuning, not just a better prompt
Before committing to fine-tuning, we spent a week trying to close the accuracy gap with better prompting alone: more detailed category definitions, few-shot examples, chain-of-thought reasoning steps. That got accuracy to about 81%, better but still short of what would let the client remove manual review. The remaining gap was in genuinely ambiguous, domain-specific distinctions that needed pattern-matching against many real examples rather than clearer instructions. That's exactly the kind of gap fine-tuning closes and prompting often can't.
Architecture
Building a real evaluation-grade dataset first
The client had roughly six years of historical claims with human-assigned categories, but not all of it was usable. Category definitions had drifted over time, some assignments were inconsistent between intake staff, and a meaningful share of older records had incomplete descriptions. We spent the first three weeks on data quality, working with the client's claims operations lead to identify and exclude unreliable labels, rather than fine-tuning on the full dataset and inheriting its inconsistencies.
Fine-tuning on the cleaned dataset
With roughly 40,000 cleaned, labeled claims, we fine-tuned a mid-size model specifically for this classification task. We deliberately didn't use the largest available model. Classification into a fixed set of eighteen categories doesn't need a frontier model's general reasoning, and a smaller fine-tuned model matched to the task gave comparable accuracy at meaningfully lower inference cost, which mattered at the client's claim volume.
Evaluation harness tied to business impact, not just accuracy
Raw accuracy wasn't the only metric that mattered: a misclassification between two water-damage subcategories with the same adjuster team was a much lower-cost error than a misclassification that routed a claim to entirely the wrong department. We built a weighted evaluation reflecting the actual operational cost of different error types rather than a flat accuracy number. It changed which model checkpoint we selected for production, because the checkpoint with the highest raw accuracy wasn't the one with the lowest weighted business cost.
Ongoing monitoring for drift
Claim patterns shift over time (a regional weather event temporarily changes the mix of claim types, for instance). We built monitoring that tracks the model's confidence distribution and flags when incoming claims look meaningfully different from the training data, an early signal that the model may need retraining before accuracy silently degrades.
Challenges and tradeoffs
- Historical label inconsistency was worse than initially estimated. Our first pass assumed about 5% of historical labels needed excluding; a deeper audit with the claims operations lead found closer to 18%, mostly from a period when one intake team had informally reinterpreted category definitions. This added roughly a week and a half to data preparation, but it was necessary: in an early experiment, training on the uncleaned set produced a model that confidently repeated the same inconsistencies.
- The cost-weighted evaluation metric took real back-and-forth with the client to define correctly. "How much worse is this misclassification than that one?" isn't purely an engineering question. It required the claims operations team's domain judgment, encoded into a cost matrix we could evaluate against, and took several working sessions to get right.
- Confidence-threshold tuning traded off automation rate against error rate directly. A higher threshold sends more claims to manual review (less time saved) but produces fewer wrong-routing errors. We tuned it with the client rather than picking a default, landing on a threshold that automates about 82% of claims while keeping the weighted error cost within their target.
Results
The fine-tuned model reached 94.1% raw classification accuracy on the held-out evaluation set, an 18-point improvement over the original prompted approach and 13 points over the best improved prompt. The cost-weighted error rate, which matters more operationally, improved by an even larger margin, because the remaining errors skewed toward low-cost confusions between closely related categories rather than department-level misroutes.
Inference cost per claim came in 62% lower than running the same volume through a frontier general-purpose model, a direct result of using a right-sized model for a well-defined classification task. The 82% of claims that clear the confidence threshold now reach the correct adjuster team the same day, instead of after a one-to-two-day manual triage delay.
What we'd do differently
We'd audit historical label consistency more rigorously before committing to a data-cleaning timeline. Our initial estimate of how much of the historical dataset needed excluding was too optimistic, and a more thorough early audit would have let us set expectations for the data-prep timeline accurately from the start.