The problem
The client is an early-stage payments startup processing card-not-present transactions for a handful of mid-size e-commerce merchants. Chargebacks were climbing, and their board had pushed hard for a fraud-scoring solution before their Series A closed. A well-known vendor in the space had already run two demos with leadership, and the deal was close to signed: an annual platform license with a minimum twelve-month commitment, priced at around $400,000 a year based on transaction volume.
We were brought in not to build anything yet, but to sanity-check the decision before the contract was signed. That's a smaller, less glamorous engagement than a full build, and one of the most valuable things we do. The client's team didn't have the bandwidth or the fraud-modeling background to verify the vendor's claims independently, and the vendor's sales engineers, understandably, weren't going to volunteer their product's weaknesses.
What we actually did
We ran a six-week evaluation, with signing paused until it finished, and one explicit goal: test the vendor's core marketing claim (a 94% fraud-catch rate at a 2% false-positive rate) against the client's own anonymized historical transaction data, not the vendor's benchmark dataset.
Weeks 1-2: getting real data into a testable shape
The client had eighteen months of labeled transaction history, with fraud confirmed through chargebacks. We spent the first two weeks cleaning and structuring it into a held-out test set the vendor had never seen. That mattered enormously: a vendor's published accuracy numbers are usually measured on data resembling what their model was tuned on, and card-fraud patterns are heavily merchant-specific.
Weeks 3-4: running the vendor's trial API against real numbers
The vendor offered a trial API for evaluation. We ran the client's held-out transaction set through it and scored the results independently. On the client's merchant mix, the vendor's precision came out at roughly 11%: nearly nine in ten transactions it flagged as fraud were legitimate. That doesn't necessarily make it a bad product. It was a mismatch between a general-purpose model tuned on a broad merchant base and this client's narrower, higher-risk category (digital goods, which fraud rings target differently from physical retail).
Weeks 5-6: a cost comparison that included the parts vendors don't put in the deck
We compared three paths: signing the vendor contract as is, negotiating a narrower vendor package with custom tuning (quoted at an additional $180k in professional services), and building a focused in-house model scoped to the client's transaction patterns. We priced the third option, including our engineering time and the client's ongoing model-maintenance cost, at roughly a quarter of the vendor's first-year total.
Challenges and tradeoffs
- Getting the vendor to run a fair trial took negotiation. The vendor's standard trial API had rate limits that would have taken months to run our full historical set through. We had to ask their team for a temporary higher-limit key for evaluation, which meant being transparent that we were running a formal build-vs-buy assessment, not just kicking the tires.
- The board wanted a yes/no answer, not nuance. "Don't sign this specific contract, but a narrower build makes sense" is a harder message to deliver than a simple recommendation either way. We built a one-page cost model showing all three paths side by side so the board could see the tradeoff themselves rather than taking our word for it.
- Internal politics around the deal already being "basically done." The executive who had run point on vendor selection was invested in that relationship. We kept the evaluation strictly quantitative so it stayed a numbers conversation rather than becoming a personality conflict.
Results
The client did not sign the vendor contract. We built the narrower fraud model in a follow-on engagement over ten weeks. It now runs in production on the client's actual transaction mix with a measured false-positive rate under 3% on live traffic, a far better fit for this client's risk profile than the vendor's model showed in testing, at a fraction of the cost.
The board's takeaway, in the client's own words afterward: the evaluation itself cost less than one month of the vendor's proposed license, and it changed a twelve-month financial commitment before it was made rather than after.
What we'd do differently
We now push for trial API access with realistic rate limits at the start of the process by default. Losing time to a rate-limited evaluation window is an avoidable delay, and we've since built this into how we scope these engagements from day one.