What production agents actually look like

If your picture of a production agent is a tireless digital coworker that runs for hours without supervision, the best available data will disappoint you. That disappointment is useful. It is far cheaper to adjust expectations now than after an outage.

Measuring Agents in Production (Pan et al., arXiv, December 2025) collected 306 survey responses and studied 86 deployed systems, including 20 in-depth case studies. Its findings read less like science fiction and more like a description of careful software engineering. Below are five of its numbers, what each one tells you, and what to do about it.

68%: runs are short

Of the production agents studied, 68% stop for a person after no more than ten steps.

What it means. Production agents are mostly short, supervised bursts of work, not open-ended missions. A long trajectory compounds error: every step is another chance to misread a tool result or drift from the goal.

What to do. Set an explicit step cap from the first prototype, and decide what the user sees when it is hit. Log the step count for every run. If the median creeps upward over the weeks, find out why before you raise the cap.

70%: off-the-shelf models

Seventy percent of teams prompt off-the-shelf models rather than fine-tuning their own.

What it means. The differentiation is mostly in the system around the model: context, tools, checks, and evaluation. Fine-tuning has its place, but it is rarely the first lever.

What to do. Keep model choice behind a thin interface so you can swap providers and rerun your test set. Spend the first engineering weeks on tool design and evaluation, not training pipelines.

74%: humans do most of the evaluation

Seventy-four percent rely primarily on human evaluation.

What it means. Many agent tasks are fuzzy, domain-specific, and costly to get wrong, so automated scoring alone is not yet trusted. LangChain's State of Agent Engineering survey points the same way, with human review (roughly 60%) slightly ahead of LLM-as-judge grading (roughly 53%).

What to do. Make human evaluation systematic rather than anecdotal. Write a rubric, have reviewers score the same sample, version the rubric, and turn every disagreement into a sharper criterion. Automate what can be automated (schema validity, forbidden tools, citations present) so human attention goes to the judgment calls.

80%: structured control flow

Eighty percent of the case studies use structured control flow rather than letting the model plan freely.

What it means. Most working systems are workflows with a model inside, or agents fenced in by code. Freedom is granted narrowly.

What to do. Draw your control flow on one page. Mark which edges are code and which are model decisions. If more than a couple of edges belong to the model, ask whether each one needs to.

85%: custom harnesses

Eighty-five percent of the in-depth case-study teams built their own agent implementation rather than adopting a third-party framework.

What it means. Frameworks are excellent for learning and prototyping. When a system has to be debugged at 2 a.m., many teams want a loop they can read top to bottom.

What to do. Whatever you build on, make the core loop legible: assemble context, call the model, parse tool calls, check them against policy, execute, log, test the stop conditions. That loop is where limits and the off switch live. If explaining a failed run means spelunking through three layers of abstraction, you have a dependency you do not control.

The number behind the numbers

Across the study, reliability was the challenge practitioners raised most often, and every pattern above is a response to it. Short runs, human review, structured flow, and owned harnesses are what reliability engineering looks like when the component at the center is probabilistic.

Compare your design to the field

Use this list as a quick side-by-side between your design and the systems that are actually running:

• [ ] A maximum step count is configured, and a test proves it triggers.
• [ ] Irreversible actions require approval or dual control.
• [ ] Tools are on an allowlist, and their arguments are schema-validated.
• [ ] Each run has a spending limit, an alert, and a way to stop it.
• [ ] A golden set of cases runs before every merge.
• [ ] Every tool call carries a searchable trace ID.
• [ ] The runbook has a section titled "How to turn the agent off."
• [ ] Security risks are written in the vocabulary of the OWASP Top 10 for Agentic Applications (goal hijack, tool misuse, memory poisoning, cascading failures, rogue agents) so the security team can review them.

A design that looks very different from the systems in the study is not automatically wrong. It should, however, be presented to stakeholders as an experiment, with the extra controls an experiment needs.

Sources cited

• Pan et al., Measuring Agents in Production (arXiv 2512.04123, submitted December 2025)
• LangChain, State of Agent Engineering (survey of 1,340 respondents, November 18–December 2, 2025; report published February 2026)
• OWASP GenAI Security Project, Top 10 for Agentic Applications 2026 (released December 9, 2025)