What are evals? Tests that check whether AI does its job well
An eval is a test that gives an AI system an input and grades its output against defined criteria. Its parts, graders and how to start with a few cases.
What is an eval?
Traditional software gives the same answer to the same input, so a test compares against an exact output. A language model can answer in different ways that are all correct. OpenAI’s documentation puts it this way: generative AI is variable, which is why standard software testing falls short. An eval does not ask “is this equal to that?”, but “does this meet these criteria?”.
OpenAI defines evals as tests that check model outputs meet style and content criteria you specify. Anthropic adds what they are for in practice: they make problems and behavior changes visible before they reach users. Without them, it says, debugging is reactive: you wait for complaints, reproduce the failure by hand and hope nothing else broke.
What are the parts of an eval?
Anthropic describes an anatomy with these pieces:
- Task: a single test with defined inputs and success criteria.
- Trial: each attempt at a task. Because outputs vary between runs, you run several trials.
- Grader: the logic that scores one aspect of performance. A task can have several.
- Transcript: the complete record of a trial, with outputs, tool calls and reasoning.
- Outcome: the final state in the environment. For example, whether the reservation actually exists in the database, not whether the agent said it did.
- Evaluation harness: the infrastructure that runs evals end to end: it executes tasks, records steps, grades and aggregates results.
The detail most often overlooked is the outcome. An agent can say “done” and have done nothing. The eval must check the real state, not the text of the reply.
Who grades the output?
Anthropic distinguishes three grader types, each with its trade-off:
| Type | Examples | Strength | Cost |
|---|---|---|---|
| Code | Text matching, binary tests, outcome verification | Fast, cheap and objective | Brittle; limited on subjective tasks |
| Model | Rubrics, pairwise comparison, natural-language assertions | Flexible and nuanced | Non-deterministic, costlier, must be calibrated against people |
| Human | Expert review, spot-check sampling, A/B tests | Reference quality | Expensive and slow |
OpenAI proposes the same split (metrics, humans and an LLM as judge), warns about automated-judge biases such as favoring position or length, and recommends validating the judge against human labels. A practical rule: code where the criterion is objective, a model where you need nuance, and people to check the judge.
Why does it matter to a designer?
Because evals turn “it feels better” into a decision. If you change the prompt of your product’s assistant, the eval tells you whether the answer to a billing question is still correct, whether the tone held and whether the format the interface expects still arrives.
Evals also speak the team’s language. A criterion like “if the user asks to cancel, the agent confirms before executing” is a design rule you can test. Without a written criterion, the discussion ends in opinions. To see where human oversight fits, read human-in-the-loop.
How do you start?
Anthropic’s advice is to start small and early:
- Begin with what you already test by hand and with failures users report. Gather 20 to 50 cases from real failures: complaints, answers you had to correct, support tickets.
- Write the success criterion for each case so that two experts in the subject would independently reach the same pass or fail verdict.
- Include cases where the behavior should occur and cases where it should not. Anthropic illustrates this with a web-search assistant: some questions call for searching (the weather) and others are answered without it (who founded Apple).
- Define how each case is graded (code, model or person).
- Run the set on every change and compare with the previous run. OpenAI calls this eval-driven development: evaluate early and often. Anthropic already said in 2024 that the key to any LLM feature is measuring performance and iterating.
- Read the transcripts. It is how Anthropic’s article ends, to confirm the graders work and the failures are fair.
Early on, small samples are enough, Anthropic explains: each change tends to have a clear, noticeable impact. But evals get harder to build the longer you wait.
Because outputs vary, one attempt is not enough. Anthropic defines pass@k as the probability of succeeding at least once in k attempts, and pass^k as the probability that all k succeed. Its example: with a 75% success rate per trial, the chance of passing all three trials is about 42%. It matters for customer-facing agents, where people expect reliable behavior every time.
The workflow OpenAI documents is similar: you describe the task, define the schema of the test data, set the graders (its “testing criteria”), run the eval on a JSONL file and analyze the results to iterate on the prompt.
Check the tool you pick too. OpenAI’s documentation says existing evals on its platform have been read-only since October 31, 2026, and that the dashboard and API are scheduled to shut down on November 30, 2026; its deprecations page points to a migration guide to Promptfoo. Keep your cases in your own files, not only inside one platform. And a public model benchmark (how to read them) does not replace an eval of your own case.
When should you NOT use one?
- When you do not know what “good” means. An eval without a clear criterion only produces meaningless numbers.
- When the case is so rare it is not worth measuring. Start with what happens every day.
- When you think an eval replaces people. Anthropic treats human review as the quality reference, even though it is expensive and slow.
Example
Illustrative example: your product has an assistant that answers billing questions. You collect 30 support questions from last month. For each one you write the criterion: “the answer states the correct deadline and promises no refund outside policy”. Two graders: a code grader checks that the correct date appears; a model grader checks that nothing outside policy is promised. You run the set three times. A question that fails two of three runs is a real problem, not an accident. Before trusting the model grader, you read ten transcripts by hand and check that its verdicts match yours.
Common mistakes
- Measuring without a written criterion. Numbers without a definition do not support decisions.
- Running one attempt and trusting it. With variable outputs, one success tells you nothing.
- Building hundreds of cases before running the first one.
- Evaluating only what is easy and leaving out the cases that hurt most in support.
- Not reading transcripts and trusting a grader you never compared with a person.
- Forgetting to update evals when the product changes. An old eval measures a product that no longer exists.
Sources
- Demystifying evals for AI agents (Anthropic Engineering, January 9, 2026): definition, anatomy, grader types, 20 to 50 tasks, pass@k and pass^k.
- Evals guide (OpenAI documentation): definition of evals and steps to create one.
- Evaluation best practices (OpenAI documentation): variability of generative AI, eval-driven development, grader types and automated-judge biases.
- Deprecations (OpenAI documentation): read-only (October 31, 2026) and shutdown (November 30, 2026) dates for the Evals platform.
- Building effective agents (Anthropic Engineering, December 2024): measuring performance and iterating on implementations.