AI feature quality metrics: what to measure and how to read it
Measure an AI feature with failure rate by category, user corrections and task outcome. How to choose quality metrics and avoid vanity metrics.
What is the difference between a model metric and a product metric?
A model metric measures the response: whether it is correct, whether its format is valid, whether it meets a criterion. A product metric measures the outcome: whether the user finished what they came to do, how many times they had to correct it and how long it took.
Both matter, but they do not say the same thing. An assistant can write impeccable sentences and still make the person ask three times for the same appointment. A model with mediocre precision can work well if its errors are easy to spot and fix. That is why it is worth measuring both and comparing them. This page focuses on output quality; for business results and adoption, see product metrics for AI.
What should you measure?
Four families, with examples of each:
| Family | What it answers | Examples |
|---|---|---|
| Failures by category | What goes wrong, and how often? | Invented date, tool not called, wrong tone |
| User corrections | How many times did the person have to fix the output? | Edits the generated text, rewrites the prompt, deletes and retries |
| Task outcome | Did the person get what they wanted? | Order completed, appointment created, file exported |
| Cost and speed | Is it sustainable? | Tokens per task, latency, cost per accepted result |
The first family comes from error analysis, which starts by reading real traces. The other three are measured in production. A useful dashboard has one metric from each family, not fifteen from the same one.
How do you build the failure categories?
Hamel Husain calls error analysis the most important activity in evals. First you read real traces and note what goes wrong in your own words; then you group those notes into categories and, at the end, count how many failures fall in each one. The count is what sets the priorities.
On sample size, his guide recommends reviewing at least 100 traces and continuing past that while you are still learning new things. If the traces come from production, selection matters: picking only those with negative user feedback misses the problems nobody reports, so it is worth combining several sampling criteria.
How do you measure a task outcome?
Anthropic’s guide to evaluating agents separates two things that are often confused. The transcript is the complete record of what happened (outputs, tool calls, reasoning). The outcome is the final state of the environment: it does not matter that the agent says “I booked the flight” if the reservation does not exist in the database. To measure quality, grade what the agent produced, not the path it took.
If the system is non-deterministic, a single attempt is not enough. Anthropic defines two measures:
- pass@k: the probability of getting at least one correct solution in k attempts. It rises with k and fits cases where one success is enough.
- pass^k: the probability that all k attempts succeed. It falls as k grows. With a 75% success rate per attempt, passing three times in a row is about 42%. It matters in customer-facing products, where consistent behavior is expected.
The same text notes that once you have evals, you can track latency, token usage, cost per task and error rates on a fixed bank of tasks, and use them as a baseline to catch regressions.
How do you read the numbers?
Four practical rules:
- Ignore scores without a definition. If a 1-to-5 scale has no written criterion for each number, it cannot support a decision. Binary verdicts with an error category can.
- Look at frequency before intuition. Husain and Shankar prioritize fixes by counting which failures appear most, not by gut feeling. The most common failure is the first candidate, even if another one is more striking.
- Cross metrics. If the failure rate drops and the correction rate rises, the model improved in one thing and got worse in another. That often means the interface is not helping the person review.
- Separate trend from noise. With few traces, a small change can be chance. Husain recommends tracking confidence intervals for production metrics and investigating when the lower bound crosses the threshold you care about.
On dashboards, Husain is skeptical: he says he has rarely found them helpful because they “place the burden on the reader.” His alternative is a short narrative: what changed, likely causes and next steps.
Why does it matter to a designer?
Because user corrections are design data. Every time someone edits the generated answer, they tell you what the interface lacked: maybe the model did not know the context, the screen did not show the source, or the format did not match what the person needed. If you read that metric next to the traces, you can design the right fix, not just change the prompt.
Measuring the task outcome also forces you to define what “finished well” means. That step is already part of product design.
When is this NOT the right approach?
- When you have no definition of success per task. Without one, any metric is a number nobody owns.
- When the dashboard has many generic metrics and none connects to a user problem.
- When the goal is to show a pretty figure to leadership. A metric that changes no decision is decoration.
Example
Illustrative example: an invoice tool uses AI to categorize expenses. Its first dashboard had a “general accuracy” metric of 91%, which looked good. When the team reviews traces, they find three failure categories: confusing subscriptions with one-off purchases (35% of errors), not recognizing foreign vendors (40%) and assigning VAT wrongly on mixed expenses (25%). Also, user corrections cluster on foreign vendors: 30% of those expenses get edited by hand. They decide to track three things: the rate of each failure per week, corrections per 100 expenses and expenses categorized without edits. After fixing foreign vendors, corrections drop from 30 to 12 per 100 in four weeks.
For the analysis method that feeds these categories, read evals as UX research. To record corrections and outcomes in production, see LLM observability. And for approve or reject verdicts, LLM-as-judge.
Common mistakes
- Choosing metrics because they are easy to compute, not because they change a decision.
- Reporting only overall accuracy, without a breakdown by error category.
- Measuring response quality and forgetting whether the task finished.
- Testing a single attempt per case when the system answers differently each time.
- Reading a small improvement without checking the sample size.
- Ignoring user corrections, which are the cheapest and most honest metric.