Trust calibration: getting users to trust AI just the right amount

What does overtrusting and undertrusting mean?

There are two opposite errors. Overtrust, or overreliance, is accepting and using incorrect AI outputs. Undertrust, or underreliance, is rejecting correct outputs because the person does not trust the system. Both are costly. The first produces errors that go unnoticed; the second makes people abandon a tool that would have helped.

Microsoft Research describes its appropriate reliance program around the same idea: balancing both errors. Its definitions are direct: over-reliance means over-trusting AI outputs and accepting them even when they are incorrect; under-reliance means not accepting or using an AI system even when it performs well. According to that same page, either can lead to poor human-AI team performance or to product abandonment.

Why does it matter to a designer?

Because trust cannot be controlled with a number. A confident-looking, well-written answer creates trust even when it is false. An answer with visible doubts can create distrust even when it is correct. The design decides which signals the person sees, and those signals weigh more than the model’s internal accuracy.

Recent research shows how hard this is. In a 2024 preprint (Li et al.), 126 participants made decisions with the help of an AI that was right 70% of the time, but whose average stated confidence was 60% (underconfident), 70% (well calibrated) or 80% (overconfident). Each condition had 42 people. In the two miscalibrated conditions, between 64% and 67% of participants judged the AI to be well calibrated, and only 26% to 29% identified the problem. The accuracy gain from AI advice was also lower when it was miscalibrated: 6.5% and 7.2%, against 11.9% when calibrated.

In a second experiment, also with 126 participants, the authors showed the calibration level. Detection rose to 74–76%, but trust in the miscalibrated AI dropped: with the overconfident AI, the share of times people switched their answer to the AI’s fell from 69.6% to 46.1%, and underreliance rose from 6.6% to 12.0%. Decision quality did not improve in any condition. In short: making calibration visible helps people detect errors, but it is not enough for decisions to get better.

Two caveats. This is a single study, with a lab task and an AI of fixed accuracy, so it is a signal, not a rule. And the practical lesson is the same as for any trust signal: test it with real users and measure what they do after seeing it.

How do you design for calibration?

Five decisions worth making on the basis of usage evidence, not intuition:

1. Show reliability by task type. A system can be very good at summarizing and poor at arithmetic. Saying where it is more reliable helps more than a single global percentage.
2. Indicate the uncertainty of each answer. Not every answer has the same certainty. See showing uncertainty.
3. Provide a way to verify. Sources, reasoning steps or a link to the original data let people check without having to believe.
4. Design for both errors. Ask what happens when the person trusts too much and what happens when they trust too little. A review reminder helps against the first; showing past cases where the AI was right helps against the second, provided it is true.
5. Measure behaviour. How many answers are accepted unchanged, how many are corrected, how many are discarded. In the experiment above, what people said they believed and what they decided did not always match, so these figures say more than a trust scale.

For failure cases, see AI errors and recovery.

When NOT to push calibration

• When the task is low risk. For a draft of an informal email, constant warnings are more noise than help.
• When the warning becomes routine. If every answer carries the same notice, people ignore it and it loses its value.
• When the problem is the data, not the interface. Showing confidence over a model with poor data only makes the poor quality visible. Before redesigning, review quality metrics for an AI feature.
• When the user cannot verify anything. If there are no sources and no way to check, showing a confidence percentage gives a false sense of precision.

Example

Illustrative example: an invoice analysis tool that extracts amounts from PDFs. Each amount appears with a mark: green if the number matches the total line of the document, yellow if the total was calculated from line items, red if it wasn’t found and was filled in by hand. The person reviews the yellow and red ones before exporting, and the system records how many yellow ones were corrected afterwards. Over time, the team finds that yellow amounts carry a rounding error for one type of invoice, and they adjust the rule. Trust was calibrated by type of case, not with one overall percentage.

Common mistakes

• Showing a confidence percentage without explaining what it means or how it was calculated.
• Using one uniform tone of certainty for every answer, even when some are doubtful.
• Designing only against overtrust and forgetting the undertrust that makes people abandon the tool.
• Trusting a trust survey without measuring what people actually do.
• Assuming a “high confidence” label replaces review for high-impact actions.

Sources

• Microsoft Research, “Appropriate Reliance research initiative” (definitions of over- and under-reliance): https://www.microsoft.com/en-us/research/articles/appropriate-reliance-research-initiative/
• Li et al., “Understanding the Effects of Miscalibrated AI Confidence on User Trust, Reliance, and Decision Efficacy”, arXiv 2402.07632: https://arxiv.org/html/2402.07632v4