The problem
The client is a payments processor whose existing fraud model, a mix of hand-tuned rules and an aging gradient-boosted classifier, had drifted into declining a meaningful share of legitimate transactions to keep fraud losses acceptable. The risk team could see the false-positive cost clearly (legitimate customers who complained or, worse, silently churned), but every past attempt to loosen the thresholds had produced a fraud-loss spike within weeks. So the model had been left alone for over a year, even though everyone agreed it needed work.
This is a distinct problem from a vendor-versus-build decision. This client had already been through that evaluation and decided to build in-house, because their fraud patterns were unusual: a concentration of transactions in a few verticals with naturally high chargeback-adjacent behavior, like ticket resale, where a general-purpose vendor model kept flagging legitimate volume as risky by default. The engineering problem was building a model precise enough to stop doing that without letting more fraud through.
What we actually did
We built a new classifier from the ground up on the client's own transaction history, treating the precision-recall tradeoff as the central design question from day one rather than something to tune once a model existed.
Weeks 1-3: feature engineering around the verticals the old model got wrong
We spent the first three weeks on the transaction segments where the old model's false-positive rate was worst, building features that captured legitimate-but-unusual purchasing patterns there, such as purchase velocity that's normal for ticket resale but would look like fraud in typical retail. That meant drawing on the client's risk analysts' knowledge of what legitimate behavior looks like in those verticals, not just training on the existing labeled dataset and hoping it generalized.
Weeks 4-7: training against an explicit cost function, not just accuracy
Rather than optimizing a single accuracy number, we trained against a cost-weighted objective the risk team helped define. A missed fraud transaction was weighted several times more costly than a false-positive decline, reflecting the client's business priorities, but not so heavily that the model reverted to the old system's overly cautious behavior. Getting the weighting right took three full retraining cycles, each evaluated on a held-out validation set that the risk team reviewed independently before the next cycle.
Weeks 8-11: shadow deployment before any real decline decisions changed
The new model ran in shadow mode, scoring every live transaction in parallel with the production system without affecting any approve or decline decision. That let us compare its calls against real outcomes (confirmed fraud, confirmed legitimate, and the ambiguous middle) before trusting it with production traffic, and report the false-positive reduction with confidence rather than relying on a validation-set number that might not hold up on live traffic.
Challenges and tradeoffs
- The risk team's institutional caution, earned from a prior bad threshold change, was a real obstacle to work through, not around. An attempt to loosen fraud thresholds two years earlier had caused a loss spike that the current risk lead still mentioned in nearly every meeting. We addressed this directly by making the shadow-mode period non-negotiable and longer than we might have used with a less risk-averse client, because rebuilding trust mattered as much as the model's performance.
- Per-vertical thresholds meant per-vertical monitoring, which the risk team wasn't originally staffed for. Moving from one threshold to several meant more dashboards and alert conditions to watch. We built a consolidated monitoring view that rolls per-vertical performance into a single weekly review, which kept the operational burden close to what the team had before.
- Labeled fraud data has an inherent lag problem that limited how fast we could validate. Confirmed fraud (via chargeback) can take 60 to 90 days to resolve, so the shadow-mode fraud-catch numbers weren't fully confirmed until well after the model was in limited production. We were explicit that the results reported at launch would need confirming once the lagged chargeback data resolved, rather than presenting an early number as final.
Results
False-positive declines dropped 61% over the twelve weeks after full production rollout, concentrated in the high-false-positive verticals the feature engineering targeted. The fraud loss rate, measured after the full chargeback resolution window, moved by 0.02 percentage points, within the range the risk team had defined as acceptable noise rather than a real increase in fraud getting through.
The client's risk lead cites the per-vertical thresholds as the reason this rollout succeeded where the earlier attempt hadn't. That attempt used a single global adjustment, which this project's shadow-mode data showed would have been wrong for at least two of the client's highest-volume verticals.
What we'd do differently
We'd build the consolidated per-vertical monitoring dashboard before shadow mode started, not in parallel with it during weeks 8 to 11. The risk team reviewed raw per-vertical numbers by hand for the first two weeks of shadow mode, which worked but was more manual effort than necessary for a period we knew would need close attention.