Evals as UX research: reading real conversations to find failures
Treating evals as UX research means reading real conversations, noting failures and grouping them before writing metrics. How error analysis works, in steps.
Why start with research instead of a metric?
It is tempting to open an evals provider, pick metrics with attractive names like “hallucination” or “toxicity” and look at the numbers. Hamel Husain and Shreya Shankar, who teach evals to product teams, warn against this. In their evals FAQ they write that generic evaluations waste time and create false confidence when used as quality measures. Their examples are helpfulness or coherence scores and text-similarity metrics such as BERTScore and ROUGE.
In another text, “A Field Guide to Rapidly Improving AI Products” (March 2025), Husain describes the risk with an image: a team celebrates a rising “helpfulness” score while users still cannot complete basic tasks. It is like optimizing a site’s load time while its checkout is broken.
The alternative is what any UX researcher would do: observe behavior, annotate it and look for patterns from the bottom up. What changes is the material. Instead of usability sessions, you have traces: the full record of what happens from a user’s query to the final response, with messages, tool calls and retrieved data.
What is error analysis?
Husain and Shankar call it the most important activity in evals, and say they have spent 60% to 80% of development time on error analysis and evaluation in the projects they have worked on. It has three steps:
- Open coding. A person reads each trace and writes free notes about any problem they see. They compare it to keeping a journal: observation first, no categories yet. They recommend starting with about 100 diverse traces and annotating at least the first 30 yourself before reviewing an agent’s suggestions.
- Grouping. The notes are grouped into a failure taxonomy and counted. The themes become concrete failure modes that can be judged pass or fail: “gave an invented delivery date”, “did not ask for confirmation before canceling”. They prefer binary labels over 1-to-5 scales because they force clearer thinking and more consistent labeling.
- Keep going until saturation. Continue reviewing while new traces still reveal failures or change the ones you have. Past 100, keep going if you are still learning.
A practical detail: at the start, note only the first failure in each trace. An early error often causes the later ones, and fixing the origin simplifies the rest.
What does it look like in a real case?
In Husain’s field guide, a team started with a spreadsheet where each row was a conversation, wrote open notes on undesired behavior and then built a failure taxonomy with the help of an LLM. Counting the frequency of each issue showed that three of them accounted for more than 60% of the problems. Date handling failed 66% of the time; after targeted fixes, success went from 33% to 95%. Husain presents these as that team’s results, not general figures.
The guide also recommends building a custom data viewer, with the full context of a conversation on one screen and one-click feedback, and calls it the most impactful investment for these teams. For a designer, it is a research tool: like an affinity board, it exists to make looking at the data cheap.
How does it apply to design?
The method looks a lot like a findings session. Each note is an observation with evidence; each theme is a pattern that deserves a product or design decision. And counting matters: without frequencies, every theme looks equally urgent.
It also changes what you design. If the analysis shows the assistant invents delivery dates, the fix may be in the interface: show where the date comes from, or ask for confirmation before promising it. The eval only measures whether the fix worked.
There is also a theoretical reason to look before measuring. In “Who Validates the Validators?” (Shankar et al., 2024), a qualitative study describes criteria drift: people need criteria to grade outputs, but grading outputs helps them define those criteria. Some criteria depend on the specific outputs seen and cannot be fixed in advance. So it pays to read first and define what “good” means afterwards.
When should you NOT do it this way?
- When you have no real traces. Synthetic data helps you get started or test rare failures, but it cannot tell you how common a failure is in production. Compare it with real data as soon as you have some.
- When nobody on the team knows the domain. A reviewer without context writes noise. Husain and Shankar say outsourcing error analysis is usually a big mistake, and recommend that one domain expert make the final quality calls.
- When the analysis stays in a document. The grouping has to end in decisions and in concrete evals.
Example
Illustrative example: a product team for an appointment service reviews 100 conversations with its assistant. The product person reads each one and writes a note about the first failure. When grouped, three themes appear: the assistant confirms appointments at closed hours (23 cases), does not ask for the time zone (14) and answers in English to users writing in Spanish (6). They pick the first one: they fix it in the prompt and in the interface, and only then write an eval for that failure mode.
For the general framework, read what are evals. To see how models are used as judges, LLM-as-judge. For the case set that comes out of this analysis, golden datasets. And for the people-based research side, how to research products that use AI.
Common mistakes
- Starting with generic hallucination or toxicity metrics because they are easy to switch on.
- Outsourcing error analysis to someone without product context.
- Noting everything you see without prioritizing the first failure in each trace.
- Grouping without counting.
- Forgetting the analysis is continuous. When the product changes, the failures change.