The problem
The client is a pharmaceutical manufacturer with decades of standard operating procedures, batch records, deviation reports, and quality documents spread across a document management system, shared drives, and archived exports. An internal team had built a retrieval-augmented assistant so staff could ask questions such as "what is the cleaning validation procedure for line 4?" In testing, it frequently cited superseded versions of procedures, which in a regulated manufacturing environment is worse than no answer at all. The project was paused until the data problem was fixed.
What we actually did
Inventory and deduplication
We started by inventorying every source: around 1.2 million files in several formats, including scanned PDFs. Content hashing removed exact duplicates, and near-duplicate detection on normalized text found the same document saved in multiple places with minor differences. Together, duplicates and superseded versions accounted for 61% of the corpus.
Versioning from the system of record
The document management system is the authoritative source for which version of a procedure is effective. We joined every document to its record there, including document number, version, effective date, status, and owning site, and indexed only effective, approved versions. Superseded versions are retained in a separate archive index that the assistant only searches when a user explicitly asks about history.
Structure-aware parsing and chunking
SOPs are highly structured: purpose, scope, responsibilities, numbered procedure steps, and tables. We parsed documents with layout awareness, used OCR with quality scoring for scans, and chunked along section boundaries so a retrieved passage is always a complete step or table rather than half of one. Every chunk carries its document number, version, section heading, and site as metadata for filtering and citation.
An evaluation set before any tuning
Quality assurance staff wrote 500 real questions with the correct document and section for each. Every pipeline change was measured against that set, which took retrieval from the original system's unreliable results to a 96% rate of finding the correct section within the top five results.
Challenges and tradeoffs
- Documents outside the system of record. Files on shared drives with no matching record were quarantined for the quality team to review instead of being indexed by default.
- Keeping it current. The pipeline listens for status changes in the document management system, so when a new version becomes effective, the old one leaves the index the same day.
- Validation expectations. We documented the pipeline, its tests, and the evaluation results in a form the client's computer system validation team could review, since the system supports regulated work.
Results
With the rebuilt data pipeline, the paused assistant relaunched on the same model it had used before. Retrieval accuracy reached a 96% top-five hit rate on the QA-written evaluation set, and no superseded procedure was cited in testing. Indexing only current, deduplicated documents also shrank the vector index by more than half, reducing its storage and query costs.
What we'd do differently
We would push harder at the start to make the system of record the only source. Most of our edge cases came from files living outside it, and resolving ownership earlier would have shortened the project.