An agent changes whenever someone edits its instructions, switches its model or re-syncs its knowledge base. Each change goes live in minutes, and none of them goes through a release process. Without tests tied to versions, nobody can say afterwards whether the agent got better, got worse, or just got different.
Closing that gap takes five steps: write the tests, freeze them, run them against the real agent, judge the results and compare versions. Each step has research behind it, and each one stops being done the moment it depends on someone remembering to do it by hand.
What a test process for agents has to do
The vocabulary comes from Anthropic's engineering post Demystifying evals for AI agents (9 January 2026): a task is one test with defined success criteria, a trial is one attempt at it, a grader scores some aspect of the result, and a transcript is the complete record of a trial. Six requirements follow from the research and the standards.
1. Start from what actually fails, and read it. In their FAQ AI Evals: Everything You Need to Know (18 September 2026), Hamel Husain and Shreya Shankar recommend starting with about 100 diverse traces, with one domain expert as the quality decision maker. Anthropic's version: 20 to 50 simple tasks drawn from real failures is a good start. A suite is only as good as the failures it came from.
2. Run each case more than once. One pass proves little for a system that answers differently each time. The τ-bench authors, Shunyu Yao and colleagues, introduced a metric called pass^k for the reliability of agent behaviour over multiple trials, and reported that even strong function-calling agents such as gpt-4o succeeded on fewer than half the tasks and, in the retail domain, on fewer than 25 percent when every one of eight trials had to succeed. Anthropic gives the arithmetic: an agent that succeeds 75 percent of the time per trial passes three trials in a row about 42 percent of the time.
3. Expect the criteria to move. In Who Validates the Validators? (Shankar and colleagues, 2024), a study of people writing evaluations for model outputs, the authors name a phenomenon they call criteria drift: people need criteria to grade outputs, but grading outputs is how they discover the criteria. A test suite is therefore revised over time, which means a result is only meaningful if you can say exactly which revision produced it.
4. Calibrate the judge. When a model grades free-text answers, it brings its own biases. The MT-Bench paper (Zheng and colleagues, NeurIPS 2023) found that a strong judge can reach over 80 percent agreement with human preferences, the same level humans reach with each other, and also documented position, verbosity and self-enhancement biases. Anthropic's guidance is that model graders should be calibrated closely against human experts; Husain and Shankar's is to measure a judge's true positive and true negative rates against human labels before trusting it.
5. Do not let the author of the tests loosen them. The obvious way to get a suite written is to have an agent write it. The risk is that an agent optimising for a green result rewrites the criteria instead of improving the system. The EvilGenie benchmark (Gabor, Lynch and Rosenfeld) tested three coding agents and reports explicit reward hacking by two of them, OpenAI's Codex and Anthropic's Claude Code. Whatever writes the cases must not also be able to run them, grade them or move the bar.
6. Keep the record. The NIST AI Risk Management Framework asks that test sets, metrics and the tools used for testing are documented (subcategory MEASURE 2.1), and that performance is measured and demonstrated under conditions similar to deployment, with measures documented (MEASURE 2.3). For systems classed as high-risk, Article 12 of the EU AI Act requires automatic recording of events over the system's lifetime, at a level that supports traceability. In practice, when an auditor asks why version 14 of an agent went live, you need the tests, the evidence, the judge and the person who decided.
Authoring: a second agent writes the first draft
Writing fifty good cases by hand is the reason most agents have none, so this step is automated. Quality and alignment builder is an agent that writes the test catalogue for another agent. You choose the target by name and optionally a focus. It reads what is actually deployed (the active definition with its steps, prompts, guardrails, memory settings, tools and any knowledge it can read) and derives the coverage that applies: positive and negative paths, multi-turn conversations, safety, grounding, tool use and side effects. A feature the target does not have is not applicable, so an agent without a knowledge base gets no grounding cases. Content it cannot read is reported as a gap, not filled in with invented facts.
Two properties keep it from marking its own homework.
The first is structural. The builder works under a credential that lasts five minutes, covers one organisation, never carries administrator authority, and is a ceiling on the caller's own rights rather than a key of its own. That grant can create and edit cases and publish a frozen set. It is refused when it tries to launch a run, cancel one, request a judge call or record a human review, and a test pins the launch, cancel and judge cases. The agent that writes the tests cannot be the one that runs or grades them.
The second is about what the builder is allowed to claim. Its final summary of what it created is assembled by code from what was read back from the system, ignoring identifiers and verdicts the model wrote itself. We added that after our own acceptance review caught an earlier version producing a fluent summary with invented identifiers and a certification nobody had performed, and, in another run, a criterion quietly loosened so that a case would pass. The loosened criterion was restored exactly, and the failing result stayed in the record.
Cases get a soundness check before and after writing: a deterministic criterion may not reject an answer that the case's own reference accepts. Facts, recommendations and refusals default to an empty criteria list plus a written reference for a judge, and an empty list never counts as a pass. The result reads "measured only", and the verdict stays not evaluable until a judge or a person decides.
Freezing the test set
Every edit to a case produces a new revision with a content hash. Publishing a test plan records the exact revisions it contains and a hash of the whole set. Later edits to a case change nothing about the test plan or about the runs made from it. To test a change, you publish another test plan.
When a run is launched, a manifest is written down: the test plan hash, the target agent's definition hash and the version that was active, the versions of any agents it depends on, the limits, and who admitted the run and when. The person launching must acknowledge that the agent may take real actions, since it runs with its real tools.
Running: a record, not a score
Each case runs for as many repetitions as you set, up to ten, with at most two attempts in flight, and every attempt gets its own conversation and memory identity. Multi-turn cases go turn by turn on one conversation. Every turn records the version of the agent that actually answered. If it differs from the version at admission, the attempt is marked as drifted and excluded from comparison.
Deterministic criteria cover eight kinds of check: exact text, contained text, absent text, JSON equality, a tool called, a tool not called, tool arguments, and a knowledge source retrieved. Every criterion is mandatory, so a failure is never averaged away. Missing evidence, such as a response withheld for privacy or an incomplete tool trace, yields not evaluable rather than a pass. A stopped run cancels future work and asks active work to stop but undoes nothing already done, and an action interrupted with an uncertain outcome is never replayed automatically.
Judging on saved evidence
A evaluator is an immutable, hashed revision that names a model connection belonging to your organisation. Re-scoring reads the evidence already stored and calls only the judge. In our production acceptance run on 9 November 2026, a test agent answered seven attempts across eight turns (8,784 tokens), the judge scored the seven, and the identifiers of the agent's runs were identical before and after judging.
Each assessment keeps the evaluator and evidence hashes, the prompt version, the model requested and the model observed with its fingerprint, and token usage with cost labelled as evaluator cost, apart from the agent's. A request carries its own identifier, so a retry cannot double-spend, and an outcome that stays uncertain is recorded as an error and never dispatched again.
Human reviews are appended with the reviewer's identity and time. They do not overwrite the automatic result. A calibration view then pairs judge verdicts with explicit human labels for one evaluator revision and reports agreement, with the limits stated: the labels cover a whole attempt, and agreement on chosen labels is not an accuracy estimate.
Comparing versions on recorded conditions
Two runs are compatible only if the target configuration, the limits, the dataset hash and the judging conditions all match. Anything else is exploratory, with the reasons listed. Within a case, results are paired by repetition. A critical regression stays visible even when the average improves, missing cases and missing evidence are listed rather than dropped from the denominator, and re-scores count as separate observations, not extra repetitions. The comparison states that it makes no causal attribution: change the agent, the dataset and the evaluator together and it says it cannot tell you which one mattered.
Audit trail
Creating, editing or archiving a project or case, publishing a test plan, launching or cancelling a run, publishing a evaluator, recording a judge assessment and recording a human review each write an entry attributed to the actor. Read together with the manifests, they answer the audit question in one pass: who wrote this test, against which revision, on which version of the agent, judged by which model under which evaluator, at what recorded cost. All of it sits inside one organisation, which also bears the cost of the builder's own inference. A tenant or a plan can switch the capability off, and when it is off, manual creation still works.
For a model swap, the failover story is the same loop run on purpose: a candidate against a baseline on a frozen set. For the standing review of a whole fleet, the monthly fleet review is where these results get read.
Limits
- Versions are observed, not pinned. A run tests whatever version is active when it is admitted. Drift during a run is detected and flagged, not prevented.
- Nothing launches a run for you. Starting one is an explicit act by a person or an authorised caller. A run is not automatically triggered by activating a new version, and nothing blocks an activation on a failing result.
- Spend is not hard-capped. Judge calls are checked against the organisation's limits first, and agent runs carry the real cost of the real agent, but the platform states that its spending checks are not a distributed ceiling.
- Some conditions are unknown. The state of the knowledge index at run time and historical model prices are not captured, and comparisons list them as limitations.
- Generated cases are not passed tests. The builder derives cases from the definition, not from your production traffic. Cases from real failures are still yours to add, and they are the better ones.
- The judge supports a bounded set of providers. OpenAI, OpenAI-compatible endpoints and DeepSeek, with structured output, through a connection the organisation owns.
Six checks for any testing system
Whatever you test agents with, these six checks tell a process from a dashboard.
| Check | How to verify it |
|---|---|
| The test set cannot change under a finished run | Edit a case after a run and confirm the old run still shows the original text and its hash |
| Every result names the agent version that produced it | Pick one answer and name the exact version from the record alone |
| Missing evidence is never a pass | Withhold a trace and confirm the verdict becomes "not evaluable" |
| Re-judging never re-runs the agent | Score a run twice and confirm the agent's run identifiers did not change |
| The author of the tests cannot run or grade them | Try to launch a run with the credential that wrote the cases |
| A comparison says when it is not comparable | Change the model and the dataset together and confirm the comparison says it cannot attribute the difference |