An agent that fails loudly is a gift. You get an error, a stack trace, a page at two in the morning. An agent that fails quietly is the real problem: a refund issued twice, a support ticket closed with a confident but wrong answer, a loop that spends your budget for six hours without anyone noticing.
Standard application monitoring was built for deterministic code. Agents call tools, hold state, retry, and choose their own path. To debug them you need to see the sequence of decisions, not just the final response. Here is what to look for before you buy.
Four questions a trace has to answer
Before you open a feature grid, decide what debugging means for your team. In practice you need four answers about any run:
- What happened? The ordered sequence of model calls, tool invocations, retrievals, and handoffs between sub-agents.
- Why did it happen? The exact prompt, prompt version, model and parameters, plus the context injected at each step.
- What did it cost? Tokens in and out, wall-clock latency, tool calls, retries, and a running dollar estimate per session.
- Can I reproduce it? Enough captured state to replay the run, or at least the failing step, without guessing.
A tool that cannot answer all four is a log viewer with better marketing. Not useless, but not observability.
Tracing: follow one run from start to finish
A trace represents one logical task; each span inside it represents a step. For an agent, spans should nest properly. The user request is the parent, each model turn is a child, and each tool call is a child of the turn that triggered it. If your sub-agent runs in a background queue and its work lands in a separate trace, you have lost the thread.
Two details separate a usable trace from a pretty waterfall chart. Context propagation, so the trace survives handoffs to sub-agents and background jobs. And session grouping, so when a customer returns three hours later and says “still broken,” the earlier trace is one click away.
Trace features worth insisting on
- Nested spans with parent-child links, including async and sub-agent work
- Full prompt and completion text, with redaction or truncation controls
- Prompt version or commit hash attached to every model call
- Replay for a single step, not only for the whole session
- Filtering by user, session, agent version, model, and status
Tool call logging: the receipts that matter
Most agent failures trace back to a tool. The model called the wrong one, passed a malformed argument, or misread a valid response. If your logs only show tool_called: search_orders, you will spend the afternoon guessing. Log both sides of the exchange, with enough structure to query it:
- Tool name, version, and the agent step that called it
- Arguments as actually sent, after validation and transformation — not the model’s raw draft
- The raw response, plus status code, error class, and retry count
- Latency split between your queue time and time inside the external service
- Idempotency key, so you can prove whether a write happened once or twice
- A pointer to the full payload wherever you truncate large responses
One habit pays off quickly: keep tool calls in a queryable store rather than a dump. “Every call to refund_payment in the last week that returned a 5xx” should take ten seconds to answer.
Catching failures before your users do
Agent failures arrive in more flavors than exceptions. Build a taxonomy and alert on the ones that hurt:
- Hard errors — API failures, auth problems, timeouts.
- Schema violations — output that does not parse against the shape you expect.
- Loop and budget breaches — step counts or costs above a threshold for one session.
- Tool misuse — calls to tools that do not exist, or arguments an endpoint keeps rejecting.
- Silent wrongness — the run completes cleanly but the answer contradicts policy, data, or the user’s request. Hardest to catch, and the reason you need sampling plus a human review queue.
Alert on rates, not single events. A 2% rise in schema violations across a version rollout is a signal; one timeout at three in the morning is noise. Watch drift too: cost per session and steps per task creeping up over weeks usually means a prompt change or a flaky dependency, not a model problem.
Auditing production behavior
Once agents touch money, customer data, or anything a regulator cares about, you need records you can hand to someone else. Ask about:
- Retention and deletion — configurable windows and a working deletion path for individual users.
- Redaction — pattern scrubbing for emails, card numbers, and tokens at ingestion, before storage.
- Access control — separation between engineers reading prompts and anyone reading user data, plus an access log.
- Immutability — audit records that cannot be edited later, exportable in an open format.
- Residency and hosting — where logs physically live, and whether you can self-host or run everything in your own cloud account.
If your agents handle personal data, run the retention and redaction plan past your legal team before you switch anything on. Retrofitting this after an incident is expensive and stressful.
The buyer’s checklist
- OpenTelemetry-compatible ingestion, or an SDK that does not force a rewrite of your agent framework.
- Nested traces with async and sub-agent support.
- Queryable tool-call logs, including arguments and responses.
- Replay for a single step and for a full session.
- Alerts on error rate, latency, step count, and cost, with sane defaults.
- Sampling controls and a human review queue for silent failures.
- Redaction, retention, and role-based access out of the box.
- Export into storage you already run.
- Pricing you can model — per trace, per span, or per gigabyte — with a clear overage story.
- A straight answer to “what happens if we leave?” Data export, and no proprietary format holding your history hostage.
Ask each vendor to instrument one real workflow during the trial. Not their demo agent — yours, with its messy tool calls and long sessions.
Start with one workflow and two weeks
Pick your highest-stakes agent, the one where a wrong answer costs money or trust. Turn on tracing and tool logging, then wait for the first confusing incident. The tool that takes you from “a user complained” to “here is the exact step that broke, and here is the fix” in under ten minutes is worth keeping. Everything else is a dashboard.
Track three numbers during the trial: time to root cause on a real failure, the share of runs you can fully replay, and cost per session as reported versus what your provider bills you. If those improve, you have found your tool. If the demo was beautiful but you are still guessing, keep looking.
Photo: PIX1861 / Pixabay
