Teaching production agents: why hands-on matters

There is a kind of AI training that produces fluent spectators. Participants leave able to explain what an agent is, name the major frameworks, and sketch an architecture on a whiteboard. Then they sit down to build one and discover they have never debugged a tool call that returned malformed JSON, never watched a context window fill up, never seen an eval fail.

Production agent skills are learned the way production engineering skills have always been learned: by building things, breaking them, and fixing them, with someone experienced nearby. That is why the Agentic Production Series is built mostly around labs in real repositories rather than lectures.

What the market is asking for

Job data supports this. Agentic Ready's August 2026 Agentic AI Jobs Index counted 2,120 open agentic roles and named MCP in 506 postings, multi-agent architecture in 366, and evals in 276. Those are skills of practice. You cannot demonstrate evals in an interview by describing them; you demonstrate them by showing a test suite that caught something.

Dexity's AI Agent Engineer 2026 analysis, based on a July 2026 scan of 390 job descriptions, found that only about 10% named a specific framework. Employers care far more about transferable capability than a particular library. Training built around one vendor's interface ages quickly. Training built around patterns, protocols, and testing transfers.

The principle: every lab should fail at least once

The most valuable moment in a lab is when something goes wrong in a way the learner did not expect, and they have the tools to understand why. Good labs are designed to produce that moment. Some failure-first exercises:

• The malformed tool result. A tool returns a partial response. Does the agent notice, retry sensibly, or invent the missing data? Learners add validation and watch the behavior change.
• The step-cap trip. Learners give the agent an ambiguous task with a low step cap, observe where it stalls, and decide whether to raise the cap or clarify the task. Usually the task was the problem.
• The injected document. A retrieved file contains instructions. Learners watch the agent follow them, then build the labeling and tool restrictions that stop it.
• The broken policy. Learners deliberately change a prompt so it violates a business rule, and the eval they wrote earlier turns the build red.
• The approval wall. An action that should require a person tries to run without one. Learners implement the approval path and its audit record.
• The budget trip. A loop retries freely until it hits a spending cap, and learners make the stop graceful.

Each of these teaches something reading cannot. Anthropic's Building effective agents explains clearly why simple, composable patterns beat elaborate frameworks. Watching your own over-engineered agent fail where a three-step workflow succeeds is what makes the lesson stick.

The shape of a good lab day

A well-structured day alternates short explanation with long stretches of building:

• Framing: the pattern, why it exists, and the failure it prevents. Fifteen minutes, not ninety.
• Guided build: everyone implements the core of the pattern in a shared starter repository.
• Break-it exercise: learners receive a failure scenario and must diagnose and fix it.
• Testing: learners write the evals that would have caught that failure.
• Capstone: apply the day's pattern to a realistic task, end to end.
• Review: walk through a few learners' traces together and compare what different choices produced.

At the end of the day, every learner has a repository they built, tests that pass, and at least one trace of a failure they fixed.

Questions for evaluating any agent training

Whether you are buying training or designing it internally, these questions separate substance from spectacle:

• What will each participant have in their repository at the end of each day?
• Where in the curriculum do things break on purpose?
• Are evaluation and guardrails taught from the first session, or covered in a final hour?
• How portable is the material if your team uses a different model provider or framework?
• When did the instructor last ship or operate a production agent system?
• Could a participant apply the capstone pattern to a real internal workflow the following week?

Measuring whether it worked

Training should change what a team ships, not just what it can discuss. Three months later, look for concrete signals: more eval suites in your repositories, design reviews that mention step caps and approval tiers, engineers who debug from traces instead of guessing. If those signals are absent, the training was a pleasant event rather than an investment.

Sources cited

• Anthropic, Building effective agents (published December 19, 2024)
• Agentic Ready (Prefactor), Agentic AI Jobs Index — August 2026 report (snapshot August 31, 2026)
• Dexity, AI Agent Engineer 2026 (published September 4, 2026; based on a July 2026 scan of 390 job descriptions)