Production-Grade Agents: Evals, Guardrails, and Deployment

The problem

The agent works in the demo. Then someone asks, "Can we put it in production?"

• You can't prove it works. There's no golden set, no regression test, and no way to know whether the next prompt or model upgrade breaks it.
• A support ticket can hijack it. One hidden instruction in a ticket or README, and an over-permissioned agent reads .env and calls home.
• Guardrails are one prompt deep. "Please don't do anything dangerous" is not a security control.
• You can't see it. No metrics on tokens, denials, or failures, and no idea what it costs per day.
• There's no off switch. No runbook, no kill switch, and no owner for residual risk.

Why this, why now: Quality is the #1 barrier to putting agents in production (32% of respondents). Most organizations have observability (89%), yet only 52% run offline evals and 37% run online evals (LangChain, State of Agent Engineering, June 2026). Evals are the most-requested skill in AI-engineer job descriptions after LLMs, appearing in 56% of them (Dexity). In a UC Berkeley-led study of deployed agents, reliability was the top challenge.

The transformation

Before: a capstone or prototype agent that works on your machine.

After one day:

• You design an eval strategy: golden datasets, deterministic checks, trajectory and tool-call evals, LLM-as-judge calibrated against human labels, and human review, both offline and online.
• You've built an automated eval harness and wired it into GitHub Actions as a release gate, alongside a read-only, deny-by-default headless Grok Build review.
• You threat-model agents against the OWASP Top 10 for Agentic Applications 2026 (ASI01–ASI10) and apply concrete mitigations.
• You harden Grok Build and custom agents in layers: permission modes, allow/deny rules (deny always wins), sandbox profiles, PreToolUse and Stop hooks, and managed policy (requirements.toml).
• You instrument agents with OpenTelemetry and track cost, latency, and failure rates.
• You know the deployment patterns: headless CI jobs, ACP servers, and containerized API services with human-in-the-loop approvals.

What you'll build

About 75% of the day is hands-on. Every attack runs against fake canary secrets, in disposable copies only.

Lab 1 · Eval Harness and CI Gate (75 min) For the Inbox Triage agent (the reference agent is provided, so Course 1 isn't required), complete a 20-case golden dataset with edge cases, a non-English email, and injection attempts. Implement trajectory checks (right tools, right order, within budget, within data scope) and calibrate an LLM-as-judge against human labels using Cohen's kappa. Prove the gate catches a model that obeys injections, then push it to GitHub Actions with a headless Grok Build review step that fails the build below threshold.

Lab 2 · Red Team and Harden (60 min) Plant prompt injections in a ticket and a README and watch an unguarded agent meet them, with a localhost "attacker" and canary values that measure any leak. Then harden in layers: deny rules, a custom strict sandbox profile with kernel-level deny globs (/.env, /.pem), a PreToolUse guard hook that fails closed, a Stop hook that requires green tests, and a confirmation step for writes. Prove that each layer independently blocks the attack. Finish by standing up a local OpenTelemetry + Grafana stack (Grok Build's external OTEL stream is labeled alpha)* and charting tokens, tool denials, and sessions.

Capstone · Ship It: Production Readiness Review (80 min build + reviews) Take an agent to production readiness: the reference Inbox Triage service, or your own Course 2 or 3 capstone. Deliver a container image, an eval gate in CI, guardrail configuration, a telemetry view, a cost budget calculated from measured tokens, a runbook with a timed kill-switch drill, and a one-page threat model mapped to OWASP ASI01–ASI10 with an owner for every residual risk. Then present GO / NO-GO / GO WITH CONDITIONS to a mock review panel.

You leave with: a production-readiness package you can adapt for your own team: eval harness, CI workflow, hardened config, threat model, runbook, and readiness checklist.

Who it's for

• Senior engineers, AI engineers, and platform/DevOps/SRE teams who must ship and operate agents
• Security engineers who need to threat-model and constrain agents with shell and data access
• Engineering managers who own the go/no-go decision
• Teams already building agents. This course stands alone.

Who it's not for

• First-time agent builders. Start with Course 1 (Tue Oct 27).
• Anyone without CI/CD or Docker basics. Both are used all day.
• People looking for a compliance certification or a framework-specific deployment class. This is engineering practice, not a compliance credential.

Live on Zoom, instructor-led, about 75% hands-on. The join link is sent 4 days before class starts. A recording is included for paid attendees.

Logistics

FAQ

What do I need to pay for besides the seat? Each student is granted a Grok token certificate worth $25 for use in that class. It covers the Grok tokens and the xAI API-key credit used during the day.

Can I take this course without the first three? Yes. It stands alone for teams already building agents. You need to have built at least one tool-using agent, know CI/CD (GitHub Actions or similar) and Docker, and understand basic security concepts (least privilege, secrets management). A reference agent is provided for every lab.

Is the red-team lab safe to run on my laptop? Yes, by design. All secrets are fake canary values, the "attacker" is a localhost listener, and attacks run only in disposable copies created by a script.

What if the model just refuses the injection? Modern models often resist obvious injections, and that's a result too. But you shouldn't rely on it. The lab includes stronger variants and an offline "naive" model that does comply. Defense in depth means the controls still work when the model doesn't refuse.

Does the sandbox work on macOS? Partly. Grok Build enforces child-process network blocking only on Linux, so on macOS that one layer is a no-op. Deny rules and the guard hook still block the attack. Linux gives the full picture.

Is Grok Build's OpenTelemetry export stable? It's labeled alpha. We use it for a local dashboard in class and teach the metric names and setup so you can evaluate it, but treat it as a preview. /usage (alias /cost) and grok usage cover token and cost tracking today.

Do you cover enterprise controls? Yes, at the policy level: pinning managed policy in /etc/grok/requirements.toml, disabling always-approve, forcing SSO and a team, OIDC for CI, folder trust, and team-level zero data retention. Pinning policy org-wide is a stretch exercise in Lab 2.

Is this course affiliated with xAI or SpaceXSI? No. This is an independent course. See the disclaimer below.

Final CTA

Walk into your next readiness review with evidence, not a demo.

Thursday, November 5, 2026 · 9:00 AM–5:00 PM ET · $750

Register and pay, $750

Questions? Use the contact form.

Delivered by Vaughn Bullard · BDArgentumAI.

Disclaimer: Grok Build is a product of SpaceXSI (formerly xAI). This course is independent and is not affiliated with, sponsored by, or endorsed by SpaceXSI or xAI. OWASP is a trademark of the OWASP Foundation; this course is not affiliated with OWASP. Product names belong to their owners. Grok Build features and subscription plans change often; course content follows the official documentation as of the class date.

Sources for claims on this page: LangChain, State of Agent Engineering (survey of 1,340 professionals, June 2026); Dexity, "AI Agent Engineer 2026: Role, Skills & Salary"; Pan et al., "Measuring Agents in Production" (UC Berkeley et al., arXiv 2512.04123); OWASP GenAI Security Project, Top 10 for Agentic Applications 2026.

• ✓ Design an eval strategy: golden datasets, deterministic checks, trajectory and tool-call evals, LLM-as-judge, and human review, both offline and online.
• ✓ Build an automated eval harness and use it as a release gate in CI.
• ✓ Threat-model an agent against the OWASP Top 10 for Agentic Applications and apply concrete mitigations.
• ✓ Harden Grok Build and custom agents with permission modes, allow/deny rules, sandbox profiles, hooks, and managed policy (requirements.toml).
• ✓ Instrument agents with OpenTelemetry and track cost, latency, and failure rates.
• ✓ Deploy agents through headless CI jobs, ACP servers, or containerized API services, with human-in-the-loop approvals.

• ✓ A production-readiness package: eval harness, CI workflow, hardened config, threat model, runbook, and checklist
• ✓ Eval harness starter, golden-set template, judge rubric, and GitHub Actions workflow templates
• ✓ Docker Compose OpenTelemetry collector stack, threat-model and readiness checklists, lab guide, and slides
• ✓ Red-team scenarios that use fake canary secrets and disposable copies only
• ✓ Live Zoom session, 9:00 AM–5:00 PM US Eastern
• ✓ Recording for paid attendees
• ✓ A Grok token certificate worth $25 for use in this class
• ✓ Certificate of completion (issuance criteria confirmed in class)

1. 9:00–12:00 · Morning
   How to evaluate an agent, then an eval harness in CI so a bad change fails the build.
2. 12:00–1:00 · Lunch
3. 1:00–2:45 · Afternoon
   Guardrails, then a red-team lab on disposable copies with fake secrets. Each layer has to block on its own. Telemetry is previewed as alpha.
4. 3:00–4:45 · Capstone
   A production readiness review: go, no-go, or go with conditions, with a kill switch, a cost budget, and a one-page threat model.
5. 4:45–5:00 · Wrap
   Series recap and next steps.

A certificate of completion is part of the day. Issuance criteria are confirmed in class. Questions go through the contact form.