Latency is a product feature for agents

Accuracy wins the keynote. Latency decides whether people keep using the product. LangChain's State of Agent Engineering survey ranks latency as the second most common barrier to putting agents into production, cited by about one respondent in five and outranked only by quality.

Users are surprisingly forgiving of an agent that says "I need a manager's approval to send this." They are not forgiving of a spinner that runs for ninety seconds to categorize a ticket. The first is a product decision. The second is an engineering accident.

Start with a latency budget, not a model choice

Teams often pick a model first and discover the latency later. Reverse the order. Decide how long each kind of task may take from the user's point of view, then design the steps to fit.

Different tasks deserve different budgets:

• Interactive: a user is waiting and watching. Aim for a first visible response within a second or two, and a complete answer within a few seconds where possible.
• Background: the user asked for something and will come back, such as a report or a batch of drafts. Minutes are fine if progress is visible.
• Human-gated: the task includes an approval. Budget machine time and queue time separately, because different levers control them.

A worked example

Here is an illustrative budget for a hypothetical interactive agent answering a customer's "where is my order?" question, before and after optimization.

Before: 14.2 seconds

After: 3.4 seconds, with first text at 2.3 seconds

What changed: a fixed workflow replaced open-ended planning for a common, predictable request; independent lookups ran concurrently; an unnecessary retrieval disappeared; a smaller model handled a step that did not need a large one; and the answer streamed, so the user saw words appear as soon as generation began.

Levers, roughly in order of impact

1. Remove steps. Every sequential model call adds seconds. The fastest step is the one you deleted, and routing common requests to fixed workflows often deletes several.
2. Parallelize independent work. Lookups that do not depend on each other should run concurrently. Take care with tools that have side effects or rate limits.
3. Right-size the model per step. Classification, extraction, and formatting rarely need the largest model available.
4. Trim context. Long inputs take longer to process; shorter, focused context speeds up every call that uses it.
5. Stream and show progress. Streaming tokens and showing intermediate status ("Checking your order…") transforms perceived speed even when total time barely changes.
6. Cache what is stable. Cache retrieval results for frequent questions, and keep prompt prefixes stable where providers support caching.
7. Move judges off the critical path. Quality checks that need not block the response can run asynchronously and flag issues afterward.

Design the wait you cannot remove

Some latency is irreducible: a slow upstream system, a long document to analyze, a required approval. Design for it rather than hiding it. Acknowledge the request immediately, say what the agent is doing in plain language, give an honest estimate, and let the user leave and come back to a finished result. A two-minute task with a clear progress message and a notification at the end feels faster than a forty-second task behind a silent spinner.

Measure the parts, not just the whole

A single end-to-end number tells you something is slow, not what. Trace each component: planning calls, each tool call, retrieval, each generation, and time spent waiting for a person. Track the median and the 95th percentile for each, because users remember the slow tail.

Approval wait deserves its own line on the dashboard. If the machine portion of a task takes four seconds and the approval queue takes four hours, model optimization is not the problem worth solving.

Latency belongs in the launch decision

Treat latency as a release criterion alongside quality. An agent that is correct but slow will be quietly abandoned, and an abandoned agent still costs money to host and maintain. Agree on budgets per task type before launch, show the per-component breakdown at the readiness review, and log a blown interactive budget as a defect rather than a nice-to-have.

Sources cited

• LangChain, State of Agent Engineering (survey of 1,340 respondents, November 18–December 2, 2025; report published February 2026)