The problem
The client runs a consumer social app. Reports were a single queue: spam, harassment, and a smaller set of hard safety cases, all mixed. Reviewers opened the item, then opened the policy, then decided which clause, if any, applied. The policy is public and specific. The time went to finding the clause, not to the judgment.
Leadership asked for automation that would 'clear the obvious ones.' The trust and safety lead had a tighter version. Obvious meant a clause the reviewer could see, on an item the reviewer could open. Account-level bans, and anything on the hard safety list, were out of scope for a model decision.
Why the clause is the unit of work
A label without a clause forces the reviewer to redo the search. A clause without the item is a number they cannot defend on appeal. The queue item we shipped has both, or it is not a clear match and it stays in the human queue unmarked.
Architecture
The policy text is the label set
We did not invent a taxonomy beside the policy. Each label is a clause id the client already publishes. If the model cannot point at a clause, it does not assign a neighbor clause to be helpful.
The item is attached, not summarized away
The reviewer sees the reported content and the clause highlight. A summary that replaces the item was rejected in review, because appeals are argued on the item.
Clear match is a threshold the team set
Only items above a precision bar, measured on a double-reviewed set, are pre-labeled. The rest are unlabelled in the same queue. Pre-label is a sort order, not an enforcement.
Bans and hard safety never take the label
Account suspension, and the hard safety categories the client listed in writing, skip the model and land in the specialist queue. The product has no path where the model closes an account.
Challenges and tradeoffs
- Policy updates lagged the index. A clause revised on Monday was still being cited in its old wording on Wednesday. We now version the policy snapshot with the decision, so the appeal shows the clause as it stood, and a nightly job refuses to label against a snapshot older than the current policy.
- Context from a thread was being dropped. A single message looked like harassment and was a quote of another post. The item payload now includes the parent the report pointed at, and the evaluation set includes quote cases.
- Reviewers distrusted labels they could not overturn in one action. The first UI made 'not this clause' a separate form. Overturn is now the same control as the decision, and the overturn is the training signal for the next review of that clause.
Results
In the pilot, 64% of reports that met the precision bar were queued with a clause a second reviewer agreed applied. The other reports were left unlabelled for the existing path. No account was banned by the model. Median time to open a pre-labeled item and record a decision was about two minutes, because the clause and the item were already on the screen.
The specialist queue for hard safety was unchanged in size and still fully human. That was the point of the scope.
What we'd do differently
We would have included quote and thread cases in the first precision set. The early bar was measured on single messages, which is the easy slice of this product.