The demo worked. Twelve steps, clean tool calls, a tidy summary at the end. Three weeks into production, the same agent fails roughly one run in five, and nobody can explain why — because nobody wrote down what "working" was supposed to mean. The distance between a good demo and a dependable system is almost always a measurement problem, not a model problem.
Define the task before you define the metric
Metrics only mean something against a stated job. "Summarize the support ticket" and "resolve the support ticket" sound related and are not. One produces text; the other closes a loop with a customer.
Write a short task contract for every agent before you score anything:
- Input: what the agent receives, including the messy parts — partial records, ambiguous requests, stale context.
- Output: the exact shape you expect, and who or what consumes it.
- Done: the conditions that make a run a success.
- Partial credit: what half-right looks like, because a lot of runs land there.
If two reviewers read that contract and disagree about whether a given run passed, it is too vague. Tighten it before you spend money on tooling.
If you cannot describe what a failed run looks like, you have not defined success — you have defined a demo.
Four metrics that carry most of the weight
Task success rate
Report it as pass or fail against the contract, plus a graded score for near misses. A single average hides everything interesting: success on short requests might sit near 95% while multi-step requests with tool calls land at 60%. Break results down by request type, input length, tool used, and whether the agent had to recover from an error. The failure taxonomy matters as much as the number. Sort failures into categories — wrong tool, bad arguments, ignored instruction, gave up early — and you get a repair list instead of a mood.
Latency
Track two clocks: time to first useful output and total time to completion. People forgive a slow start more easily than a slow finish, and a background agent has different tolerance than a chat interface. Measure the tail too. A three-second median with occasional ninety-second outliers feels broken to the users who keep hitting the outliers.
Cost per action
Add up everything a run consumes: model calls, retries, tool or API charges, retrieval, and human review time. Retries are the usual culprit. An agent that quietly loops four times per success can look fine per token and terrible per completed task. Set a ceiling per task and alert when a run crosses it.
Hallucination and grounding checks
For any claim drawn from retrieved material, verify it against that material. A workable method: pull the factual claims out of an output, check each against the source it cites, and count unsupported claims per run. Pair that with a refusal check — cases where the correct behavior is "I don't have that information." An agent that never refuses is not confident, it is ungrounded.
If your agent calls tools, add two supporting measures: tool selection accuracy (did it pick the right tool) and argument validity (did the call parse and match the schema). A surprising number of "reasoning failures" turn out to be one of those two.
Benchmarks are a starting point, not a verdict
Public benchmarks tell you whether a model can do something in general. They say very little about your traffic. Scores can be inflated when test data overlaps with training data, and the tasks rarely match your input distribution. Use them to shortlist, then build your own set.
A small evaluation set that earns its keep looks like this:
- Collect 50 to 200 real cases from logs, including the ugly ones.
- Label expected outcomes by hand, with two people reviewing a sample to catch disagreements.
- Freeze the set and version it. Changing test cases mid-comparison invalidates the comparison.
- Keep a holdout you never tune against.
- Add a regression case every time you fix a bug. Bugs recur.
Where you cannot grade with rules, use human review. If you automate judging, audit it against human labels regularly — automated judges tend to reward longer answers and can be fooled by confident phrasing.
Red flags that show up in weak evaluations
- Averages with no distribution. "92% success" means little without the failure spread.
- Success defined as "it responded." Producing output is not completing a task.
- Retries hidden from the cost sheet. The bill arrives later, usually as a surprise.
- Demo-driven development. Handpicked examples presented as evidence.
- Test cases leaking into prompts. The agent passes because it has seen the answers.
- No failure categories. If every failure is "the model was dumb," you have no roadmap.
- No escalation path. Agents touching money, health, or legal outcomes need human review triggers, a cost limit, and a clear stop condition.
- Nothing measured after launch. Behavior drifts as inputs, tools, and models change.
Build the measurement loop
Start narrow. Pick three metrics — usually task success, cost per action, and one grounding or tool-accuracy check — and instrument them this week. Assemble a fifty-case set from real logs, run it before and after every change, and spend twenty minutes a week reading failures out loud with whoever owns the product. That habit catches more problems than any dashboard, because the people closest to the work notice patterns before the numbers do.
Then revisit the contract. Tasks change, inputs shift, and a metric that made sense in March may be measuring the wrong thing by June. Rewrite it when that happens. Measurement is not a one-time setup; it is the part of the job that keeps the rest of it honest.
Photo: AlphaTradeZone / Pexels
