The problem
The client is a regional property carrier. The coverage desk spends its morning on questions that repeat: is this peril listed, which deductible applies, does a prior loss change the form. Each answer has to point at a clause. The team was typing those answers from PDFs, and a routine question often sat for half a day because the right form was buried in an endorsement stack.
A general chat tool had been tried on the same PDFs. It answered quickly and it sounded sure. On a review of 80 answers, reviewers found a clause citation on fewer than half, and several answers stated a coverage position the form did not contain. That was worse than the queue. A wrong coverage sentence is a claim the carrier may have to stand behind.
Why a cited answer, not a faster chat box
The desk did not need a model that could talk about insurance in general. It needed a draft that a reviewer could check in one pass: the question, the form and endorsement that were in force, the clause, and a sentence that did not go past that clause. If the corpus did not contain the answer, the correct output was a refusal, not a plausible paragraph.
Architecture
The in-force form, not the whole archive
Policies had been refiled. An answer against a superseded form would cite a real clause and still be wrong. The retrieval set for each question was limited to the form and endorsements in force on the date of loss, pulled from the policy admin system, not from a folder of every PDF the carrier had ever stored.
A citation the reviewer can open
Every released sentence carries the form identifier, the clause heading, and the page. The reviewer opens that page. The model is not allowed to emit a coverage sentence with an empty citation. That rule is checked before the draft reaches the queue, not left as a prompt suggestion.
A refusal that is a finished answer
When the retrieved pages do not address the question, the draft says so and lists what was searched. Those items go to a senior reviewer with the gap named, instead of a fluent guess the senior then has to unwind.
Evaluation on citation and overreach
We scored held-out questions two ways: did the cited page support the sentence, and did the sentence add a position the page did not state. A fluent answer that failed either check was a miss, even if a reader would have found it helpful.
Challenges and tradeoffs
- Endorsement order was the real retrieval bug. The first index treated the base form as enough. On a sample of disputed files, the controlling language was in an endorsement that replaced a paragraph of the base form. We changed retrieval so the in-force endorsement wins, and we re-scored the evaluation set after that fix.
- Reviewers did not trust a score they could not open. An early version showed a confidence number and hid the page. The desk ignored it. Showing the page, and refusing when there was no page, is what got the draft used.
- Date-of-loss filtering was easy to get slightly wrong. A renewal effective at midnight was being matched to the prior term for losses reported that morning. We fixed the boundary with the policy admin team and added it to the evaluation set so it cannot regress quietly.
Results
On the held-out set, 96% of answers that the assistant was willing to release cited a clause a reviewer agreed supported the sentence. The other questions were refusals. None of the released drafts in that set stated a coverage position with no citation. Median time for a reviewer to accept or edit a draft was 41 minutes, against a prior typical wait of about half a day for the same class of question.
The desk still owns the letter. What changed is the search. Reviewers spend the time on the judgment, not on finding the form.
What we'd do differently
We would have sampled endorsement-controlled files in the first evaluation set, not the second. The base-form index looked accurate until we deliberately picked files where the endorsement was the contract.