Document pipeline replaced a manual triage queue

The problem

The client runs freight brokerage operations, coordinating shipments between carriers and shippers. Every load generates a stack of documents (bills of lading, proofs of delivery, carrier invoices, customs paperwork for cross-border loads, and rate confirmations), arriving by email, fax-to-email gateways, and a handful of carrier portals in wildly inconsistent formats. A team of six opened each document by hand, identified what it was, extracted the load number and key fields, and routed it into the right system and queue.

At roughly 1,100 documents a day, this manual triage was the hard ceiling on how many loads the operations team could process. The work wasn't difficult, but it was repetitive and error-prone at that volume. Misrouted documents were the single largest cause of billing disputes with carriers: a proof of delivery that sat in the wrong queue for two days looked, from the carrier's side, like the broker was stalling payment.

Why this wasn't a simple OCR problem

The client's first attempt, before engaging us, was an off-the-shelf OCR tool that extracted text from PDFs. It worked for reading text, but it didn't solve the actual bottleneck. Someone still had to look at every extracted document and decide what it was and which load it belonged to, because formats varied by carrier, and even the same carrier didn't format a given document type consistently.

The real problem was classification and routing under a lot of format variance, not text extraction. We scoped the build around that distinction early. It changed the architecture from "add OCR" to "build a document understanding pipeline that classifies first, extracts second, and routes third, with a defined fallback for anything it isn't confident about."

Architecture

Ingestion and normalization

Documents arrive through three channels (an email inbox, a fax-to-PDF gateway, and API pulls from two carrier portals), and the first stage normalizes all of them into a common format: PDF pages rendered to images, alongside any available native text layer for born-digital PDFs. This mattered more than expected, because scanned faxes and portal-generated PDFs needed different preprocessing to give the downstream vision model usable image quality.

Classification

A vision-language model classifies each page into one of the six document types the client handles (bill of lading, proof of delivery, carrier invoice, rate confirmation, customs form, other). We tuned and evaluated it against roughly 3,200 labeled real documents from the client's archive. We didn't fine-tune a model; a strong general vision-language model with a structured few-shot prompt and layout-aware preprocessing hit the accuracy target without the time and data a fine-tune would have needed.

Multi-page documents (a rate confirmation is often three to five pages) are classified per page first, then grouped, because a single incoming file sometimes bundled several unrelated documents scanned together. That was one of the more common real-world messes the system had to handle.

Extraction

Once classified, each document goes to a type-specific extraction schema: for a bill of lading, the load number, carrier, origin, destination, and weight; for a proof of delivery, the load number, delivery timestamp, and whether a signature is present; and so on. Extraction uses structured output constrained to each schema, so the model can't put a load number in the wrong field, and illegible fields are marked as missing rather than filled with a guess.

Routing and confidence handling

Every extraction carries a per-field confidence score. Above threshold, the document routes automatically into the matching load record and queue. Below threshold on any required field (most often the load number, since it's the key that links everything else), the document goes to a human review queue with the model's best-guess extraction pre-filled, so a reviewer corrects it rather than starting from scratch.

Challenges and tradeoffs

• Fax quality was the biggest accuracy drag. A meaningful share of carrier documents still arrive as faxes, and fax-to-PDF image quality varies enormously by sender. Early testing showed accuracy nearly 15 points lower on fax-sourced documents than on native digital PDFs. A dedicated preprocessing step for the fax channel (deskew, contrast normalization) recovered most but not all of that gap, and fax documents still go to human review slightly more often by design.
• Multi-document bundles required real engineering, not a prompt tweak. Detecting where one document ends and the next begins within a single scanned file was harder than classifying an individual document. We built a page-boundary detection step ahead of classification, using layout signals (letterhead changes, page numbering resets) rather than relying on the vision model to infer boundaries implicitly.
• The confidence threshold trades accuracy for review-queue volume directly. We tuned it with the client's operations lead rather than picking a number ourselves. Too conservative, and the automation doesn't meaningfully reduce the workload; too aggressive, and misrouted documents recreate the billing-dispute problem we were hired to fix. We landed on a threshold that sends about 9% of documents to review, which the client considers an acceptable manual load.

Results

Document throughput per operations staff-hour rose 4.2 times over the fully manual baseline, measured across the first eight weeks after launch. For the roughly 91% of documents that clear the confidence threshold automatically, time from arrival to correct routing dropped from same-day or next-day to under a second.

Classification accuracy on the held-out evaluation set was 96.4%. More importantly for the client, the failure mode behind billing disputes (documents landing in the wrong load's queue) dropped to near zero, because low-confidence extractions now go to review instead of being auto-routed on a bad guess. In the client's internal tracking, carrier billing disputes attributable to document handling fell by roughly two-thirds over the following quarter, though that number reflects several factors beyond this system.

What we'd do differently

We'd have asked for a larger, more representative fax-document sample earlier in discovery. We under-sampled fax-sourced documents in our initial evaluation set relative to their real share of volume, so the fax-quality gap didn't show up until we were deep into the build. That cost about a week and a half of preprocessing rework that could have been designed in from the start.