The problem
The client is a mid-size insurance company that had built an internal claims-triage model to route incoming claims to the right handling track (fast-track for straightforward claims, full review for complex ones) based on details submitted at intake. The model had passed the company's internal model-risk review and been approved for a phased production rollout. But the client's chief risk officer wanted an independent audit first, because the team that built the model was the same team that had reviewed it: a structural conflict she was uncomfortable with, however competent the team was.
It's a reasonable instinct that many companies skip. An internal team reviewing its own model, however skilled, tends to test for the failure modes it anticipated while building it and not to think adversarially about the ones it didn't, because anticipating your own blind spots is intrinsically hard. The client wanted someone with no stake in the approval looking for exactly the risks the internal review wasn't positioned to find.
What we actually did
We ran a three-week independent audit that treated the model as an adversarial target. Rather than checking whether it worked on the cases it was designed for, we probed for conditions where it might fail in ways that mattered.
Week 1: understanding the model's actual decision boundary, not its documented one
We started by testing edge cases in the model's input space that its documentation didn't address, rather than starting from its stated design intent. That's where we found the first issue: claims with a specific combination of claim type and policyholder tenure were being fast-tracked at a rate their complexity didn't justify. Nobody on the internal team had tested that combination because it wasn't one they'd thought to check.
Week 2: probing for training-data-driven bias the internal review hadn't surfaced
We ran a structured fairness analysis of claim outcomes by policyholder geography. The client's claims volume, and therefore its training data, was heavily concentrated in a few states, and we wanted to know whether fast-track routing behaved consistently for claims from less-represented regions. It didn't. Claims from the three lowest-volume states were routed to full review at a meaningfully higher rate than complexity alone would predict. That was the second finding, and it carried real regulatory exposure, given how closely insurance regulators scrutinize geographic discrimination.
Week 3: adversarial input testing and documentation gap review
We tested the model with deliberately unusual but plausible claim submissions, including intake patterns designed to probe its behavior outside the typical training distribution. That surfaced a third issue: a narrow input pattern where the model's confidence score stayed high even as its actual routing accuracy, measured against historical outcomes, dropped noticeably. The fourth finding was a governance gap rather than a model behavior. The approved documentation didn't disclose that the model had been retrained twice since its original approval, each time on an updated data cut, with no re-review of the kinds of risks we were finding.
Challenges and tradeoffs
- The geographic routing disparity required careful framing to avoid an unfair characterization of intent. The internal team hadn't built this behavior deliberately; it emerged from a training-data imbalance nobody had checked for. Our report was explicit about that distinction, because conflating "found a bias in the outcome" with "the team was careless" would have turned a legitimate, fixable finding into an unnecessarily adversarial internal conversation.
- The undisclosed retraining finding was, in some ways, the most consequential, and the hardest to present without it reading as an indictment of the review process itself. Two retrainings without re-review wasn't really a model problem; it was a gap in the governance process, since nothing in the approval workflow required re-review after a retrain. We framed it as a process recommendation (any retraining triggers re-review) rather than a failure specific to this model, because the same gap applied to every model the company built.
- Confidence-score miscalibration on out-of-distribution inputs is a subtle failure mode that's easy to underweight relative to more visible risks. A model being wrong is a familiar risk; a model being wrong while reporting high confidence is more dangerous, because it defeats the human-review safety net meant to catch low-confidence cases. We rated this finding high-severity because the rollout plan relied on confidence scores to decide when a human should double-check the routing, so this failure undermined the very safety mechanism the plan depended on.
Results
The audit identified four risks the client hadn't known about, from the confidence-miscalibration issue (rated high-severity and blocking for the phased rollout) to the retraining-governance gap (a process finding needing a policy fix rather than a model fix). The client paused the rollout after the audit report, addressed the calibration issue and the geographic routing disparity with the internal team over the following six weeks, and made re-review after any retraining a standing policy.
The chief risk officer has since made an independent adversarial audit, by a team with no stake in the model's approval, a required step before production rollout for every customer-facing model the company builds. She cites this audit as the reason for the policy change: the internal review that approved the original model was competent, but it wasn't structurally positioned to find what an outside adversarial review found.
What we'd do differently
We'd request the model's full retraining history on the first day of future audits, rather than discovering it during the week-three documentation review. It didn't change this audit's outcome, but retraining history is baseline information for any model-risk audit, and requesting it upfront makes audits more consistent.