How to Research AI Products: Calibrating Trust and Testing Failures

What makes it different?

Three things change compared with a product built on a fixed flow, according to Koji’s research guide (Koji is a research tool, so read it as a vendor’s point of view):

1. The output varies. Two people can get different answers to the same question, so a test cannot reproduce exactly what the user saw.
2. The experience is a conversation. Heuristics designed for fixed screens do not describe a question-and-answer dialogue well.
3. “Correct” depends on context. The same error means different things to different roles, and that must be explicit before you measure anything.

That is why a classic usability test can give a false sense of solidity. A flow can finish without any click problems and still leave the user with a wrong answer they never checked. Google’s People + AI guidebook adds that what users consider an error is closely tied to their expectations of the system, so it pays to ask them what they expected.

What is trust calibration?

It is the match between how much a user relies on the system and how reliable the system is on that task. Google PAIR sums it up: users should not implicitly trust the system in all circumstances, but calibrate their trust. There are two opposite errors:

A design analysis by Tian Pan describes a typical pattern: at first users over-trust, trust drops sharply after a visible failure, and then people overcorrect toward distrust, even on tasks the model handles well. It is a blog post, not a study, but it matches what matters for research: measuring that path over time, not a snapshot. For the concept from the design side, see trust calibration.

What questions should the study answer?

Four questions organize the research (they come from Koji):

• Adoption: what job the user hires the AI for, and what made them start using it.
• Calibration: when they verify an answer and when they accept it as is.
• Failure tolerance: which kinds of error break trust, and how many errors it takes before they disengage.
• Control: how much they want to steer the result, and where they want more or less agency.

How do you test failures?

A practical method:

1. Test with the real model, not recorded answers. Koji is blunt about it: do not use mocks or pre-recorded outputs, because the variation is part of what you are testing.
2. Use real tasks from the flow, not generic requests such as “try the AI,” and end each task with a short debrief.
3. Include tasks where the output is wrong. If the real model does not fail during the session, prepare a scenario where it does and check whether the user notices.
4. Observe detection: does the user notice the error, how long does it take, and what do they do next?
5. Repeat over several rounds. Google PAIR recommends planning multiple rounds of testing, because trust signals and explanations are hard to get right the first time.

When the product already has usage, Koji proposes “failure-mode replay” interviews: you start from production logs, find a failure pattern (a type of hallucination, a cluster of refusals) and interview the people who experienced it. Useful questions: did they notice, did it hurt their work, did they tell anyone, and do they still use the feature. Google PAIR adds that some errors are never reported, the “background errors” where neither the user nor the system registers the failure, so you need your own quality checks on top of user feedback.

How do you measure calibration?

Measure in three layers, because self-reporting alone is not enough:

• What the user says: reported trust on short scales and open questions.
• What the user does: according to Koji, the share of outputs they accept as is, edit or discard; how often they check outputs against another source; how many bad outputs they tolerate before they stop using it; and whether they come back after a failure. Tian Pan adds abandonment, how often people bypass the feature, and time to override.
• The relation between both: whether trust matches the model’s accuracy on that task.

Showing the model’s confidence is a delicate matter. Zhang, Liao and Bellamy (2020) found in two experiments that confidence scores helped people calibrate their trust, but better calibration did not guarantee better decisions. Tian Pan also warns that language-model confidence scores are often miscalibrated, and showing them that way can worsen calibration. Google PAIR suggests asking whether the figure changes any user decision, testing different formats (categories such as high, medium or low, or percentages with context), and leaving it out if it adds nothing.

How do you recruit for an AI product?

Three criteria matter more than age or location:

1. Prior experience with similar tools: a new AI user reacts differently from someone who already suffered an error with another assistant.
2. Domain expertise: whoever pays the invoices, makes medical decisions, or manages inventory catches errors that a general audience overlooks.
3. Real diversity: Google PAIR recommends including people who reflect your audience because, for example, familiarity with probabilities varies widely.

If the team can identify them from its records, include people who already experienced a failure too. They contribute the most about how trust is rebuilt.

When should you NOT use this approach?

• When the product does not yet have a real model to test: the results would be about something else.
• When you do not know what counts as an error in the domain. Define that with experts first.
• When you only measure general satisfaction: it hides the loss of trust in specific tasks.
• When the study uses people who would never actually use the product.

Example

Illustrative example: a team designs an assistant that summarizes invoices for small businesses. It tests with the real model and prepares three scenarios where the total comes out wrong. It finds that half the participants accept the summary without checking the total, especially when the assistant looks very confident. The team changes the interface to prompt a check on figures and tracks the correction rate in production for a month.

What they learned: the failure was not in the summary itself, but in how the interface presented the model’s confidence.

Common mistakes

• Testing with simulated or recorded answers.
• Measuring only satisfaction and not trust in specific tasks.
• Showing the model’s confidence without first testing how users read it.
• Using general participants for a specialized domain product.
• Considering calibration solved after a single session.
• Expecting users to report every failure: some pass unnoticed.

For the next step, how to design recovery when the system fails, read AI errors and recovery.

Sources

• Koji, User Research for AI Products: A Practical Guide for 2026 (vendor content)
• Tian Pan, The Trust Calibration Curve: How Users Learn to (Mis)Trust AI (blog)
• Google PAIR, People + AI Guidebook: Explainability + Trust
• Google PAIR, People + AI Guidebook: Errors + Graceful Failure
• Zhang, Liao and Bellamy (2020), Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making (arXiv)