Offline evals in CI: the merge gate demos miss

Most agent teams can tell you what went wrong yesterday. Fewer can stop it from going wrong tomorrow. LangChain's State of Agent Engineering survey quantifies the gap: observability is reported by about 89% of respondents, offline evaluation by only about 52%. Observability is a camera. An offline eval in CI is a gate.

This post is a practical guide to building the smallest gate that works, in roughly a week of engineering time.

Step 1: Choose forty cases that hurt

Do not start with a public benchmark. Start with your own pain. Pull candidates from:

• support tickets where the agent gave a wrong or embarrassing answer;
• near misses caught by a reviewer;
• the tasks your stakeholders care about most;
• adversarial inputs, such as an instruction embedded in a document;
• edge cases: empty search results, a tool timeout, an ambiguous request.

Forty well-chosen cases beat four thousand generic ones, because each encodes something your team has already learned the hard way.

Step 2: Write each case as data

Store cases in the repository next to the agent code, in a simple format. An illustrative example:

{
  "id": "refund-over-limit-003",
  "input": "I was charged twice, please refund $480.",
  "fixtures": {"account": "acct_basic_tier", "charges": "dup_charge_480.json"},
  "expect": {
    "outcome": "escalate_to_human",
    "must_call": ["lookup_charges"],
    "must_not_call": ["issue_refund"],
    "max_steps": 6,
    "cite_policy": true
  },
  "severity": "hard"
}

Look at what this checks. Not only the final answer but the path: which tools must and must not be called, how many steps are acceptable, whether policy was cited. For agents, the route matters as much as the destination.

Step 3: Run headless with controlled tools

The runner executes each case against the agent with tools replaced by fixtures or sandboxed versions. Fixtures make runs repeatable and cheap; a sandbox exercises real integration code. Use fixtures for most cases and a handful of sandboxed runs for integration confidence.

Because models are not fully deterministic, run each hard case more than once (three repeats is a common choice) and require every repeat to pass.

Step 4: Separate hard fails from soft scores

Hard fails encode policy: things that must never happen. Soft scores encode quality: things that should improve over time. Blending both into one averaged number hides the hard fails, which is exactly backwards.

For soft scores, use exact match where answers are structured, rubric grading by a model where they are not, and periodic human calibration of that grader. The same LangChain survey found model-as-judge grading at about 53% and human review at about 60%, so the two methods are complements rather than rivals.

Step 5: Wire it into the pipeline

A minimal CI job, sketched for a typical hosted pipeline:

agent-evals:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - run: pip install -r requirements.txt
    - run: python -m evals.run --suite golden --repeats 3 --out results/
    - run: python -m evals.gate --results results/ --hard-fail-max 0 --soft-drop-max 0.05
    - uses: actions/upload-artifact@v4
      if: always()
      with: { name: eval-traces, path: results/traces/ }

Two details matter. First, trigger the suite when prompts, tool definitions, retrieval settings, or the model version change, not only when application code changes. Second, publish traces of failing cases as artifacts so the developer can see what happened without reproducing it locally.

Step 6: Give it an owner and a feedback loop

An eval suite without an owner decays. Assign one person, ideally someone on the agent's on-call rotation, to review new cases, retire obsolete ones, and keep thresholds honest. Make one rule non-negotiable: every production incident adds at least one case before it is closed.

What offline evals will not do

They will not catch drift in live traffic: new kinds of questions, changes in upstream data, seasonal behavior. That is the job of online evaluation, which samples production runs and scores them asynchronously, and which only about 37% of surveyed teams practice today. Build the offline gate first, because it is cheaper and stops the regressions you cause yourself, then add online sampling.

A five-day plan

Monday, collect cases. Tuesday, write them as data. Wednesday and Thursday, build the runner and fixtures. Friday, wire the gate and break something on purpose to watch it fail. Teams that do this rarely go back, because a red build on Thursday is far cheaper than an apology on Monday.

Sources cited

• LangChain, State of Agent Engineering (survey of 1,340 respondents, November 18–December 2, 2025; report published February 2026)