Most teams ship an AI agent with functional testing (does it do the task) and no real evaluation harness (does it keep doing the task correctly as the system, the data, and the model change over time). Those are different things, and skipping the second one is how a system that worked well at launch quietly gets worse without anyone noticing until a user complains.
What an evaluation harness actually is
A structured, repeatable way to measure an agent's performance against defined metrics on a labeled dataset, run regularly, not just once before launch. The point isn't to prove the system works on day one. It's to catch the moment it stops working as well as it used to, whether that's from a model version change, data drift, or a new edge case in real usage that your original test set never covered.
What to actually measure
Task accuracy, scored against real examples, not synthetic ones
A labeled set built from actual production cases (with real messiness included) is a meaningfully better predictor of real-world performance than a clean synthetic test set. This is the same principle behind good adversarial security testing: your evaluation is only as good as how representative your test cases are of what actually happens.
Correct tool use, measured separately from final-answer correctness
An agent can arrive at a plausible-looking correct answer while having called the wrong tool, or called the right tool with subtly wrong arguments. Measuring tool-use correctness as its own metric, not just whether the final output looked right, catches failures that would otherwise stay invisible until they cause a real problem.
What the agent correctly declines to do
Just as important as measuring what an agent does correctly is measuring whether it correctly recognizes when it shouldn't act: when a request is ambiguous, out of scope, or would require authorization it doesn't have. This is a distinct evaluation category from task accuracy, and it's the one most teams skip, even though it's often the difference between a safe system and a costly incident.
Cost and latency, not just correctness
A technically accurate agent that costs three times what it should, or responds too slowly to be usable, has failed just as surely as one that gives wrong answers. Track these alongside accuracy, not as a separate concern measured only when someone notices a large invoice.
Building a real evaluation dataset
The most common failure in building an evaluation harness isn't the metrics; it's the dataset. A representative labeled set, built from real historical cases with real domain-expert review, takes genuine time to assemble correctly. We've found this step alone can take a meaningful share of an evaluation project's total time. It's not time you can skip, because evaluating against an unrepresentative dataset produces a confident but wrong signal about how the system will actually perform.
Running it continuously, not just once
An evaluation harness that only runs before launch tells you nothing about drift afterward. Running it on a regular cadence against fresh production samples, and specifically flagging when performance shifts meaningfully from baseline, is what turns a one-time QA gate into an actual early-warning system for the kind of silent degradation that's otherwise invisible until a user notices.
How we approach this
We build evaluation datasets from real, representative examples, measure tool-use correctness and appropriate refusal separately from task accuracy, and run evaluation on an ongoing cadence, not just as a pre-launch checkbox.