Observability without evals cannot say no

Observability has become the default first investment in agent infrastructure, and that is a good thing. LangChain's State of Agent Engineering survey found it in place at roughly 89% of responding organizations. Yet in the same survey quality remained the leading barrier to production, and online evaluation, the practice of scoring live traffic, sat at only about 37%.

That combination describes a lot of teams: they can see everything their agents do and still cannot prevent the next bad run. A dashboard can show you a failure. It cannot refuse one.

Four questions, four practices

It helps to be precise about what each practice answers:

• Tracing answers what happened in this run? It records every model call, tool call, retrieval, and decision.
• Metrics answer how often, how fast, how expensive? They aggregate across runs.
• Offline evals answer did this change make things worse? They run before deployment against a fixed set of cases.
• Online evals answer is live behavior drifting? They score a sample of production runs after the fact.

Observability covers the first two. Only evals can say no, and the two kinds guard different doors. An earlier post covered the offline merge gate in detail; this one is about the bridge between what you observe and what you test.

Instrument for evaluation, not just debugging

Traces become far more useful when they are designed to feed evaluation. For each span, capture enough to replay and score:

• Run level: task type, user or tenant (pseudonymized), model and prompt version, total steps, total cost, outcome.
• Model calls: input token count, output, latency, and the version of the context-assembly code.
• Tool calls: tool name, validated arguments, result status, latency, and whether an approval was required and granted.
• Retrieval: the query, document IDs returned, and their freshness and authority status.
• Guardrails: every check that ran, whether it passed, and what it blocked.

Redact personal data at capture time, not later. Traces that cannot be shared safely with the team will not get looked at.

If you use OpenTelemetry, model these as spans with consistent attribute names across services, so agent traces sit alongside the rest of your system's telemetry rather than in a separate silo.

From trace to test: a weekly loop

The practice that separates mature teams is a routine that turns observations into evaluations. A simple version:

1. Sample. Each week, pull production runs: some at random, some flagged by users, some flagged automatically (high cost, high step count, guardrail fired, low judge score).
2. Score. Reviewers grade the sample against the team's rubric. Where an automated judge is used, compare its scores with human grades on the same runs to confirm the judge is still calibrated.
3. Promote. Every confirmed failure becomes a new case in the offline eval set, with its trace attached.
4. Track. Chart online scores by week and task type, and alert when a task type drops past an agreed threshold.
5. Review. Spend fifteen minutes of a regular team meeting on the trend and the new cases.

The survey shows human review (about 60%) and model-graded evaluation (about 53%) both in wide use. In this loop they cooperate: people set and calibrate the standard, and the automated judge applies it at scale.

Decide what to keep, and for how long

Trace storage gets expensive, and keeping everything forever is a privacy liability. A tiered policy works well: keep full traces for a short window (long enough to investigate a typical complaint), keep sampled and flagged traces longer, and keep traces linked to eval cases for as long as those cases exist. Write the policy down, align it with your data-retention rules, and make sure the cases in your eval set never point at a trace that has already been deleted.

Alerts that matter

Most agent dashboards alert on errors and latency. Add alerts for the signals that tend to precede quality problems:

• a sudden rise in refusals or "I can't help with that" responses;
• step counts or cost per run climbing above their usual range;
• a spike in guardrail activations, which may indicate an attack or a bad upstream change;
• falling citation rates on tasks that should be grounded;
• rising approval reject rates for a task type.

Each of these points to a trace you can open and, often, a test case you should write.

A test for your own team

Ask whoever owns your agent to show you two things: the trend line of eval scores over the past month, and the most recent production failure that became a test case. If they can show both, observability is feeding a control system. If they can only show a dashboard, you have a very good camera and no brakes.

Sources cited

• LangChain, State of Agent Engineering (survey of 1,340 respondents, November 18–December 2, 2025; report published February 2026)