Create tests with Quality and alignment builder — Operator manual | AI Agent Builder

Create tests with Quality and alignment builder

Prepare reviewable agent test catalogs from the agent configuration.

2 min read · Updated November 9, 2026

Quality and alignment helps compare an agent’s behavior against your goals, rules, and knowledge through tests and evidence. Creating cases does not establish alignment: results must be reviewed after execution.

Quality and alignment builder prepares a test catalog from an agent you can access. Choose the agent; you do not need to write an initial prompt. It reads the available configuration and authorized resources, proposes coverage, and distinguishes observed facts from missing evidence.

Create a catalog

  1. Select the organization that will own the tests and pay for the model usage.
  2. Open Monitoring → Quality and alignment → New project → Create with Quality and alignment builder.
  3. Choose the Agent to evaluate. Leave Focus (optional) empty for general coverage, or specify a concern such as safety, factual accuracy, or tool usage.
  4. Select Analyze and propose tests. This prepares a proposal without writing a catalog or executing the target agent.
  5. Review the coverage and gaps. Use chat to ask for more cases or a different focus.
  6. Select Save catalog once. The builder creates the cases, reviews their references, applies scoped corrections, and reads back the saved revisions and immutable test plan.
  7. Continue through Open project and cases or View test plans and execution options in the existing evaluation screens.

Cases can contain a single message or an ordered conversation. Review reference answers for completeness and accepted alternatives. Natural-language facts do not require exact wording; a case with no deterministic criteria is measured rather than automatically passed. Execution, semantic judging, and human calibration are separate actions.

Availability and consumption

An administrator can enable or disable Quality and alignment builder in the organization's feature flags or plan. An ancestor restriction or installation license can also limit availability. When disabled, manual evaluation authoring remains available to authorized users, while builder entry, direct navigation, resolution, and new runtime requests are blocked.

The shared builder belongs to AI Agent Builder. Model connections, usage, and generated catalogs belong to the selected consumer organization, under its existing permissions and limits. No owner credential is needed.

Interrupted saves

If the connection is interrupted, use Verify save to reconcile the existing project. Do not assume nothing was written or submit another Save blindly. New conversation clears the current proposal and saved-resource references; it does not delete catalogs.