The problem
The client is a commercial real estate firm that regularly acquires multi-property portfolios, and every acquisition requires lease abstraction: pulling structured data (rent schedules, renewal options, escalation clauses, termination rights, and a dozen other fields) out of every lease in the portfolio. That's often dozens or hundreds of documents, each formatted differently depending on which law firm drafted it and when. Analysts read each lease by hand and entered the terms into a spreadsheet. For a mid-size portfolio this took roughly six weeks and created real deal-timeline pressure, because due-diligence windows don't wait for abstraction to finish.
The documents themselves were the hard part. Leases in a typical portfolio spanned twenty years of drafting conventions across dozens of law firms: no consistent document structure, inconsistent terminology for the same legal concept, and a meaningful share of older scanned documents that needed OCR before extraction could even start.
What we actually did
We built an extraction pipeline that combines OCR for scanned documents, a document-structure classifier to handle inconsistent formatting, and a field-extraction model tuned on commercial lease language. It produces structured output with a confidence score and an exact source citation for every extracted field.
Weeks 1-2: building a training set from the firm's own document variety
We worked with the client's analysts to hand-label a representative sample of their document variety, deliberately weighted toward the oldest and worst-formatted leases rather than the cleanest recent ones, because a model that only handled modern, well-formatted documents wouldn't solve the actual problem. The labeling, done by the analysts whose time the tool was meant to save, took real coordination, but it meant the model was validated against exactly the documents it would face in production.
Weeks 3-4: OCR and structure-aware extraction, not flat text extraction
For scanned documents, we used a layout-aware OCR pipeline that preserves document structure (which clause a piece of text belongs to, not just raw text in reading order). A rent-escalation figure means something different in the base rent clause than in an operating-expense clause, and flat text extraction had shown a meaningfully higher error rate in early testing precisely because it lost that context.
Weeks 5-6: confidence scoring and citation, then analyst review workflow
Every extracted field ships with a confidence score and a highlighted citation pointing to its exact location in the source document. A review interface lets an analyst quickly confirm high-confidence fields and focus on low-confidence ones instead of re-reading every lease. We deliberately didn't build an auto-accept threshold letting high-confidence fields skip review, even though the accuracy numbers would have supported it for the top tier. The data informs acquisition pricing, so even a small residual error rate on unreviewed data was an unacceptable risk for the client.
Challenges and tradeoffs
- Structural inconsistency across twenty years of law-firm drafting conventions was the dominant source of extraction errors, not OCR quality itself. We initially assumed poor scan quality on older documents would be the main accuracy bottleneck, but early error analysis showed that structural ambiguity (which clause a term belonged to) caused more mistakes than OCR errors. That shifted effort in weeks 3-4 toward structure-aware extraction rather than further OCR tuning.
- Analysts were, reasonably, worried the tool was built to replace their jobs, and that needed direct handling, not just reassurance. We involved the same analysts who did the labeling in designing the review workflow, and positioned their role explicitly as verification and judgment rather than data entry. The client's operations lead reinforced that framing in how the tool was rolled out internally.
- A small number of lease clauses used genuinely ambiguous language that no extraction model, however well-tuned, could resolve with confidence. For clauses where even the human labelers disagreed on interpretation, we built an explicit "flag for legal review" output rather than forcing the model to produce a confident-looking answer to a genuinely ambiguous question.
Results
Lease abstraction for a portfolio acquisition, previously a six-week process, now takes roughly two days including analyst review, measured across three acquisitions completed since the tool went into production. Field-level accuracy against analyst-verified ground truth held at 94.6% across the full field set, and the review workflow concentrates analyst attention on the lower-confidence tail instead of requiring uniform scrutiny of every field.
Every field sent to the acquisition-pricing team carries a source citation and confidence score. The deal team credits this with making faster abstraction trustworthy to acquisition committee members who weren't involved in building the tool and need to verify a number quickly during a pricing discussion.
What we'd do differently
We'd weight the initial labeling even more heavily toward structurally inconsistent older documents from the start, rather than adjusting the sampling mix once early error analysis showed that structure, not scan quality, was the harder problem. We corrected course quickly, but a sample built with that lesson in mind would have saved a few days of re-labeling.