Evals are the skill hiding in your job descriptions

Read a stack of agentic engineering job postings and a pattern emerges. The headline skills are protocols, orchestration, and retrieval. Somewhere further down, often in a single line, sits "experience with evaluation." That line is doing more work than it looks.

What the postings say

Agentic Ready's first Agentic AI Jobs Index, a snapshot taken in June 2026, counted 2,072 open agentic roles across 221 companies. Evals appeared in 219 of those postings. That is fewer than MCP (454) or multi-agent architecture (369), but evaluation is the skill that determines whether the others can be trusted. An engineer who can wire up five tool servers but cannot tell you whether last week's prompt change made the agent worse is shipping risk, not capability.

What teams actually do

LangChain's State of Agent Engineering survey shows how wide the gap is in practice. Observability is close to universal among respondents, at about 89%. Offline evaluation sits at about 52%, and online evaluation at about 37%. Quality is the most-cited barrier to production.

In other words, most teams can watch their agents. Far fewer can stop a regression before it ships. That is the skill the job descriptions are reaching for, often without naming it precisely.

Rewriting the line in the job description

"Experience with evaluation" invites résumé keywords. A more useful version describes the work:

You will own the golden test set for our agents: deciding which cases belong in it, wiring it into CI so regressions block merges, and turning production incidents into new test cases within a week.

That sentence tells candidates what the job is, and it tells your own team what you expect to exist.

Five layers a strong candidate can explain

When you interview for this skill, listen for fluency across five layers, not just final-answer scoring:

1. Outcome checks. Did the agent produce the right result? Exact match where possible, a rubric where not.
2. Trajectory checks. Did it get there legitimately? Forbidden tools, excessive steps, loops, and skipped approvals count as failures even when the answer is right.
3. Tool correctness. Were arguments schema-valid? Were write calls blocked when they should have been?
4. Groundedness. Do the cited sources actually support the claims?
5. Adversarial and budget cases. Does the agent refuse injected instructions and resist goal hijack, the first risk listed in the OWASP Top 10 for Agentic Applications? Does it stay within its cost and time limits?

A candidate who only talks about layer one has evaluated models. A candidate who talks about layers two through five has evaluated agents.

An interview exercise that works

Skip the whiteboard algorithm. Instead, hand the candidate a small agent (a loop, three tools, and ten logged runs) and ask:

• Which of these ten runs would you call failures, and why?
• Write three test cases that would have caught the worst one.
• Which of your cases should block a merge, and which should only be tracked as a trend?
• If the model provider ships a new version next month, how would you decide whether to accept it?

Forty-five minutes is enough. You will learn more about judgment than from any framework quiz, and the candidate learns what your team values.

Starting your own golden set

If your team is among the roughly half without offline evals, start smaller than you think:

• Collect 30 to 50 tasks that would genuinely hurt if the agent got them wrong. Pull them from real tickets, complaints, and near misses.
• For each, write the expected outcome and at least one trajectory rule, such as "must not call send_email."
• Run the set on every change to prompts, tools, or model version.
• Make one person the named owner of the set, and treat edits to it like edits to code: reviewed, versioned, explained.
• After every production incident, add a case before closing the ticket.

Within a month, the set becomes the team's shared definition of "working," which is worth more than any single score it produces.

Three early mistakes

• Averaging everything into one score. A single number hides the one case where the agent issued an unapproved refund. Keep policy violations as separate pass/fail checks.
• Grading only with another model. Automated judges scale well, but they drift and share blind spots with the system they grade. Calibrate them against human scores on a sample every few weeks.
• Freezing the set. A golden set that never changes measures last quarter's product. Retire cases that no longer reflect real use, and say why in the commit message.

Teaching it

Evaluation is teachable but rarely taught well, because it is less exciting than building. The fastest way to learn it is to break something on purpose and watch a test catch it. Any training or onboarding you invest in should include that moment. If people leave able to build an agent but not to prove it works, the most important line of the job description is still unfilled.

Sources cited

• Agentic Ready (Prefactor), Agentic AI Jobs Index — June 2026 report (snapshot June 23, 2026)
• LangChain, State of Agent Engineering (survey of 1,340 respondents, November 18–December 2, 2025; report published February 2026)
• OWASP GenAI Security Project, Top 10 for Agentic Applications 2026 (released December 9, 2025)