What an agent CI gate must check before a prompt or model change ships

The problem: a regression with no failing test

An AI agent can get worse while every unit test stays green. Someone edits the system prompt, rewrites a tool description, or moves to a newer model ID. The code compiles and the tests that mock the model pass. Then the agent starts issuing a refund before it looks up the order, or calls the same write tool twice, or says it escalated when it did not.

So the question for a platform team is practical. How do we stop a prompt, model or tool change from shipping a regression, and what must the CI gate check?

Anthropic's guidance says automated evals are "especially useful pre-launch and in CI/CD, running on each agent change and model upgrade". The hard part is the design: what to grade, which grader to trust, how many times to run each case, and what blocks a merge versus only warns.

LangChain's "State of Agent Engineering", a vendor-run public survey of 1,340 respondents that likely skews towards LangChain's own audience, reports that 89% have observability for their agents but 52.4% run offline evals, and that quality is a top barrier for 32%. Treat that as directional. Seeing what an agent did after the fact is not the same as stopping a bad change before it ships.

What the gate should check, in order of cost

It helps to split "the agent works" into layers, cheapest first. This layering is engineering reasoning built on the sources, not a published standard.

Anthropic notes that latency, token usage, cost per task and error rates "can be tracked on a static bank of tasks", which is why budgets belong in the same suite rather than in a separate dashboard. The same post warns that "One-sided evals create one-sided optimization": for every case where the agent should take an action, keep a case where it should not. An agent that escalates everything will pass a suite that only tests escalation.

Final answer or trajectory

One option is to grade only the final answer. The other is to score the whole path: which tools, which arguments, in which order.

Anthropic's post warns about the second. Checking that agents followed very specific steps is "too rigid and results in overly brittle tests", and "it's often better to grade what the agent produced, not the path it took." Note "often". It does not say never check the path. AWS Prescriptive Guidance lists "component-level insight", assessing every step and not just the result, as a trait of a robust evaluation framework.

These fit together once you ask why a path matters. The reasoning: grade the outcome first, and check the trajectory only where the path carries risk.

A refund without a lookup on the same order, a write before an approval, a second side effect. Those are safety properties, so they block. "Which three read tools did it call, in what order" is usually style, and a test for it breaks the next time a better model finds a shorter route.

So prefer in-order or any-order matching, keep exact order for the few flows where order is the point, and write most path rules as policy checks ("no refund without a prior lookup of the same order ID") rather than full expected sequences.

Deterministic checks versus LLM judges

Anthropic recommends "choosing deterministic graders where possible", and is clear about the cost on both sides. Code-based graders are brittle to valid variations. Model-based graders require calibration with human graders. It also suggests giving an LLM judge a way out, such as returning "Unknown", instead of forcing a guess.

AWS Prescriptive Guidance adds that an LLM judge needs validation against ground truth, may inherit the judge model's biases, is less reliable for highly subjective or safety-critical tasks, and should be audited against human-evaluated data periodically for drift.

Shopify's engineering blog describes its judges moving from "barely-better-than-random (Cohen's Kappa of 0.02)" to "near-human performance (0.61 vs. human baseline of 0.69)". Those are Shopify's numbers on Shopify's data. The general point is that an uncalibrated judge can sit close to random, and you only find out by measuring.

The reasoning: block on deterministic checks, and warn on LLM-judge scores until the judge has measured agreement with human labels on your data. An unchecked judge can block good changes and pass bad ones, and a judge model update can move scores when the agent did not change.

One trial or k trials

Model outputs vary between runs. Anthropic's post says they "run multiple trials to produce more consistent results", that each trial should start from a clean environment, and that shared state between runs can cause correlated failures. With more than one trial, you must define "pass". pass@k "measures the likelihood that an agent gets at least one correct solution in k attempts". pass^k "measures the probability that all k trials succeed", and it "especially matters for customer-facing agents where users expect reliable behavior every time."

An agent that refunds correctly in four of five trials is not safe to ship for that case. The reasoning:

• Use pass^k for critical cases. Every trial must pass.
• For the rest, use a pass-rate floor plus an allowed per-case drop from a committed baseline.
• Do not rerun a red gate until it goes green. That quietly turns pass^k into pass@k.
• Keep three outcomes: pass, fail, and "could not evaluate" (model call errored, telemetry missing, span truncated). Never count the third as a pass. Only errored trials may be retried.
• Do not rely on temperature 0 to remove variance.

With five trials, one failure moves a case by 20 percentage points. That is arithmetic, not a measured result, and it is why per-case thresholds depend on the trial count.

The recommendation: a two-tier gate

The layering is reasoning; the individual pieces come from the sources above.

Tier 1, on every pull request. Run the cases against a stubbed or recorded model, with deterministic contract, policy, risky-trajectory and budget checks over the agent's spans. It is fast, has no model cost and gives the same result every run. It catches regressions in agent code, tools, schemas, orchestration and the harness. It cannot catch a prompt or model regression, because the model is not called.

Tier 2, nightly and whenever a pull request touches the system prompt, tool descriptions, model ID or inference settings. Run the same cases against the real model with k trials. Anthropic recommends "extensive testing in sandboxed environments, along with the appropriate guardrails". For live evals this matters most, because real tools can issue real refunds.

Notice that only Tier 2 calls the real model, so a prompt or model regression can only show up there, and that AgentCore Evaluations scores the same agent again on live traffic after release.

Gate rules for both tiers: zero trials fails; errored trials fail with their own message; any failed trial on a critical case fails; a pass rate below the floor fails; a baseline case that did not run fails, so deleting a hard case is visible; a case dropping more than the allowed amount below baseline fails.

An illustrative config:

# Illustrative only. Values are placeholders, not tested thresholds.
tier1:
  trigger: every_pull_request
  model: recorded            # scripted or recorded responses, no live calls
  trials_per_case: <k_stub>
  block_on:
    - deterministic_policy_checks
    - risky_trajectory_checks
    - output_schema
    - tool_call_budget
  gate:
    zero_trials: fail
    errored_trial: fail
    critical_case: all_trials_must_pass   # pass^k
    min_pass_rate: <floor>
    missing_baseline_case: fail
    max_case_drop: <allowed_drop>

tier2:
  trigger:
    paths: [prompts/**, tools/descriptions/**, config/model.*]
    schedule: nightly
  model: live                # sandboxed tools and data only
  trials_per_case: <k_live>
  block_on: [same as tier1, token_budget_increase]
  warn_on: [llm_judge_scores]          # until judge agreement is measured
  rerun: errored_trials_only

How each kind of check should behave across stages, again as reasoning:

Where the cases come from

Anthropic says "20-50 simple tasks drawn from real failures is a great start", and that user-reported failures should become test cases. AWS Prescriptive Guidance says to build golden datasets partly from production or pilot logs with poor outcomes. Shopify describes moving towards "Ground Truth Sets" built from sampled production conversations.

The working rule: every support escalation, low online-eval score or incident becomes a case, with expected trajectory and outcome, before the fix merges. Keep the baseline in Git so a removed case shows up in review.

Anthropic also says regression evals "should have a nearly 100% pass rate", while capability evals start low and can "graduate" into the regression suite. A set that never grows stops finding new failures.

What breaks in production

Reasoning unless a source is named:

• A stub-only gate gives false confidence. It proves the checks work, not the model.
• Judge drift: a judge update changes scores with no agent change. AgentCore built-in evaluator configurations cannot be modified, so record which evaluator ran.
• Telemetry gaps: AgentCore code-based evaluators take at most a 6 MB payload, and session spans may be truncated when the whole payload is larger. Missing tool arguments make a check unevaluable, which is its own failure.
• Moving conventions: the OpenTelemetry GenAI agent span and event conventions, including gen_ai.evaluation.result, showed "Status: Development" when checked on 2026-10-16. Checks that parse span attributes need fixture tests.
• Live cost grows as cases times trials times turns. Gate on tokens per task.
• Locked evaluators: an enabled AgentCore online evaluation config locks its code-based evaluators until disabled or deleted, so changing a production check needs a plan.

Where AgentCore Evaluations fits on AWS

AgentCore Evaluations became generally available on 2026-03-31. The announcement says on-demand evaluation "enables teams to test agents programmatically, supporting regression testing in CI/CD pipelines", while online evaluation samples and scores live traces. It supports built-in, custom LLM-based, and Lambda-hosted code-based evaluators in Python or JavaScript.

The docs describe three types. Online runs on live traffic with sampling or filters. On-demand runs on span or trace IDs you pass and suits build-time testing. Batch is an asynchronous job over CloudWatch Logs sessions, designed for baselines and pre and post comparison after prompt or model updates.

Later release notes add A/B testing on live traffic and Failure Insights, in public preview, which finds recurring failure patterns, including "silent behavioral failures that produce no error signal".

Mapped to the two tiers:

• Risky trajectory checks: Builtin.TrajectoryExactOrderMatch, Builtin.TrajectoryInOrderMatch and Builtin.TrajectoryAnyOrderMatch use programmatic scoring, no LLM calls and zero tokens. Builtin.Correctness and Builtin.GoalSuccessRate use LLM-as-a-Judge, so they start in the warn column.
• Policy checks: code-based evaluators can run the same policy logic as your CI checks, on-demand and online, within a 5-minute Lambda timeout.
• Tier 2 and pre-release: the on-demand dataset runner is documented for "dev-time iteration, CI/CD pipelines, and small datasets". It calls your agent through an invoker you supply (AgentCore Runtime, a direct function call or HTTP), waits a configurable delay (default 180 seconds, paid once per run) for CloudWatch ingestion, collects spans and calls the Evaluate API, mapping expected_response, assertions and expected_trajectory. So it needs a running agent that sends supported telemetry to AgentCore Observability, AWS credentials in CI and an ingestion wait per run. It does not replace Tier 1.
• After release: online evaluation and A/B tests, with low scores fed back into the case set.

One limit: custom evaluators with ground-truth placeholders cannot run in online evaluation, so not every pre-release check carries into production. On coverage, the July 2026 release notes list OpenAI Agents, LlamaIndex, Google ADK and Claude Agent SDK alongside Strands and LangGraph, plus generic support for OpenTelemetry GenAI or OpenInference telemetry.

The managed option fits best when the agent already runs on AgentCore or emits supported telemetry, and the team wants one evaluator set before and after release. A self-hosted Tier 1 should stay cheaper and faster for every pull request either way; that is reasoning from how the two run, not a measured comparison.

When a different approach is better

• No agent yet, or a single LLM call: Amazon Bedrock Evaluations covers models and knowledge bases, including external models and RAG sources. Its overview does not list agents or tool trajectories, and that is fine here because there is no trajectory to check.
• Read-only agents with no side effects: outcome-only evaluation is usually enough. Trajectory checks add brittleness without guarding anything.
• Highly subjective output, like tone or writing style: human review on a sample beats an automated block. AWS guidance says LLM judges are less reliable for highly subjective tasks, and in the LangChain vendor survey, 59.8% of respondents reported using human review.
• Hard safety rules, like never refunding above a limit: enforce them at runtime, outside the agent. An eval samples behaviour; it does not prevent it. Keep the eval as a second line.

A reference build, and what it does not show

InfraTales/agent-evals-ci is a reference build of the gate logic above. It is not a production system.

Its CI runs a strict eval gate against a scripted model. There are 8 eval cases, 5 of them critical, and the gate applies all six gate rules above, with minPassRate as the pass-rate floor and maxCaseDrop as the allowed per-case drop. On every push to main and every pull request, CI runs npm audit and npm run check: types, lint, tests, the stub eval suite with the gate, and cdk synth with cdk-nag. The latest CI runs succeeded.

What it does not show:

• It has not called a real model. The live-evals workflow has 0 runs and is skipped until an AWS role variable is set, and there is no live baseline. The README says: "Nobody has deployed this stack or run live mode against a real account yet."
• It is not deployed.
• The prompt-injection case uses a pre-scripted safe model response and has no case-specific expectations, so in stub mode it only shows that a safe path passes the shared policy checks. It does not show that any model resists injection, and it does not forbid a single refund after a lookup.
• It has no cost gate yet. The only budget enforced is a tool-call budget of 6; there is no token, cost or latency budget.
• Its 16-mutation script is not part of CI.

In other words, it shows the Tier 1 shape running green against a scripted model, and the Tier 2 path written but not run.

Sources

Pages were read on 2026-10-16.

Amazon Bedrock AgentCore Developer Guide

• Evaluation types
• Ground truth evaluations
• Evaluators
• Custom code-based evaluator
• On-demand dataset runner
• Release notes for Amazon Bedrock AgentCore

Other AWS documentation

• Amazon Bedrock AgentCore Evaluations is now generally available, AWS What's New: general availability.
• Evaluate the performance of Amazon Bedrock resources, Amazon Bedrock User Guide: the scope of model and RAG evaluation.
• Evaluating quality and reliability in generated outputs, AWS Prescriptive Guidance: judge validation and dataset guidance.

Standards and specifications

• semantic-conventions-genai, OpenTelemetry: the status of the GenAI conventions.

Industry sources

• Demystifying evals for AI agents, Anthropic Engineering: CI evals, dataset size, pass@k and pass^k, trial isolation, grader choice, trajectory brittleness, regression versus capability suites, and budgets.
• Building Effective AI Agents, Anthropic Engineering: the sandboxing advice.
• Building production-ready agentic systems: Lessons from Shopify Sidekick, Shopify Engineering (company engineering blog): the judge agreement figures.
• State of Agent Engineering, LangChain (vendor-run survey): the survey figures.

Reference code

• InfraTales/agent-evals-ci, GitHub: own reference build.