The problem
The client sells online with a published return policy and an order system that already knows what was delivered and when. A large share of support chats were returns: is this eligible, what is the window, where is the label. Agents re-read the policy and the order, then typed the same answer. A first chat that hedged produced a second chat, because the customer still did not know if the label was coming.
The team had an order-status assistant already. Returns were a different policy with a different cost of being wrong. An exception given in chat became a social post when the warehouse refused it.
Why the policy and the order, together
Eligibility is the policy applied to this order, not the policy in the abstract and not the order in the abstract. The assistant may state an outcome only when both are present: the clause, and the order fact it applies to. If either is missing, the chat goes to a person with what was found.
Architecture
The policy page is the clause source
We index the published return policy, including the exceptions the page already lists. A clause that exists only in an internal macro is not used, because the customer cannot be held to a rule the site does not show. Where the macro and the page disagreed, the page won and the macro was fixed.
The order supplies the facts
Delivery date, item category, and whether the item was marked final sale come from the order. The assistant does not ask the customer to retype a fact the order has, and it does not let a customer assertion override a final-sale flag.
The outcome is a template tied to both
Eligible, ineligible, and needs-a-person are templates. Each names the clause and the order field. A free-form exception, a goodwill refund the policy does not mention, is not a template. Those chats transfer.
A resolved chat is one that does not come back
The metric the client already used was a second contact on the same order within seven days. We kept it. A chat that sounds finished and generates a follow-up is not a resolution.
Challenges and tradeoffs
- Final-sale items were described as final in the catalog and not on the order. The assistant trusted the order, and the order was wrong. We now require the flag on the order line, and items missing the flag transfer instead of being treated as returnable by default.
- The policy window was measured from delivery by the page and from shipment by the macro. Customers were quoted both. We set the assistant to the published page and the client updated the macro. The evaluation set includes an order on each side of the window.
- Label creation failed silently. The chat said a label was on the way and the warehouse job had failed. The template now waits for the label id, and without it the chat says the label is not ready and routes to a person.
Results
In production, 58% of return chats closed with no second contact on that order within seven days, against the client's baseline for the same chat type. Reviewed outcomes all quoted a policy clause and an order field. Chats that needed an exception were transferred, and the transcripts did not show the assistant granting one.
Agents still handle the edge of the policy. They see fewer chats whose only need was the sentence the policy page already contained.
What we'd do differently
We would have reconciled the macro and the public page before indexing either. We indexed the page, which was right, and still spent a week explaining answers that matched the page and not the macro agents had memorized.