The problem
The client is a mid-size commercial legal practice handling a steady volume of vendor and partnership contracts for corporate clients. A junior associate's first pass on a new contract meant manually checking every clause against the firm's internal playbook: acceptable liability caps, required indemnification language, non-standard termination clauses, and a dozen other categories the firm had refined over years of negotiation. That first pass routinely took three to five hours on a moderately complex contract, almost all of it spent scanning for deviations from the firm's standard positions rather than on the judgment calls that actually needed a lawyer's expertise.
The firm's partners were clear from the first conversation that they had no interest in an AI system that reviewed contracts and gave a final answer. What they wanted was something that did the scanning work, flagged exactly where a contract deviated from the playbook and by how much, and left every judgment call and every sign-off with an attorney.
What we actually did
We built an agent that reads an incoming contract, checks it clause by clause against a structured version of the firm's playbook, and produces a flagged, annotated first-pass review that an associate works from instead of starting cold.
Weeks 1-2: turning an unstructured playbook into something checkable
The firm's playbook existed as a mix of internal memos, precedent language in past contracts, and knowledge that several senior partners carried in their heads. We spent the first two weeks working with two partners to convert this into a structured rule set: roughly 40 clause categories, each with acceptable ranges (for instance, a liability cap between one and three times contract value) and known problematic variants the firm had negotiated around before. This structuring work, not the agent's model logic, turned out to be the highest-leverage part of the engagement.
Weeks 3-5: building the review agent and its evidence trail
The agent reads the contract and classifies each clause against the playbook's categories. For every deviation it produces a specific citation (the exact contract language), the playbook rule it deviates from, and a severity flag. We deliberately built it never to draft a replacement clause or negotiation position, only to identify and characterize the deviation, because generating counter-language was closer to practicing law than the firm wanted an AI system doing at this stage.
Weeks 6-7: validation against attorney judgment before rollout
Before any associate used the agent on a live matter, we ran it against sixty already reviewed historical contracts, using the firm's own attorney annotations as ground truth. We measured recall (did it catch what the attorney caught?) and precision (did it flag things that weren't actually deviations?) separately by clause category. Overall recall was strong at 97%, but it was weak on one category: non-standard termination language buried in definitions sections rather than in the termination clause itself. We fixed this by adding a definitions cross-reference pass before rollout.
Challenges and tradeoffs
- Getting partners to agree on ambiguous playbook rules took longer than building the agent itself. Several clause categories had inconsistent internal guidance, with different partners applying slightly different standards in practice. We ran a structured reconciliation session with the firm's managing partner to resolve these before encoding anything. An agent enforcing a rule the firm didn't actually agree on would have been worse than having no rule at all.
- The 3% recall gap on edge-case termination language was a real limitation we didn't try to talk around. Even after the definitions cross-reference fix, a small number of unusual clause structures still slipped past the agent in later spot-checks. We were explicit with the firm that the agent is a first-pass tool, not a substitute for an attorney's full read, and the rollout materials say so directly rather than implying full coverage.
- Associates needed a workflow change, not just a new tool, and adoption took real internal effort. Several associates initially treated the agent's output as optional extra reading rather than the starting point for their review, which defeated the time-saving goal. The firm's own training lead, not us, ran onboarding sessions reframing the flagged output as the first thing to open, and that mattered more for adoption than anything in the build.
Results
Median first-pass review time on a moderately complex contract dropped from roughly four and a half hours to about thirty minutes, a ninefold reduction. This was measured across the first two months of firm-wide use, against the same complexity band of contracts used in the pre-launch baseline. Attorney-measured recall against the firm's playbook held at 97% in the two months after rollout, consistent with pre-launch validation; the managing partner cited that figure when presenting the results to the full partnership.
No contract has been approved without a licensed attorney's review and sign-off. That was a deliberate design constraint from day one, not something the firm had to enforce afterward, and the firm considers it as important to report as the time savings.
What we'd do differently
We'd run the partner reconciliation session for ambiguous playbook rules before starting any build work, rather than in parallel with early development. We ended up revising two clause categories mid-build once the session surfaced disagreements we hadn't anticipated. It was manageable, but better sequencing would have avoided it.