Explainability in AI products: how much to explain and how

Explainability shows users why the AI said something. How much to explain depending on risk, how to present confidence and what to do if reasoning is hidden.

info Quick Definition
Explainability in an AI product is the information the system shows users to understand why it produced a result, what data it depends on and how reliable it is. Its goal is not to open the model’s black box but to give the person what they need to decide when to believe it and when to review it.

What should a product explain?

There are four questions users ask, sometimes without saying them: why this came out, with what data, how sure the system is and what I can do if it is wrong. Explainability answers those questions to the extent the case requires.

Google PAIR’s People + AI Guidebook says every AI prediction is based on data, so data sources must be part of the explanation: what data is used, whether the result is personalized or aggregated, and whether the person can remove or reset it. That part is not technical; it is part of the relationship with the user.

NIST (the US National Institute of Standards and Technology), in its September 2021 report NISTIR 8312, proposes four principles for explainable systems that work as a review checklist: the system gives evidence or reasons for its outputs (explanation); those reasons are understandable to whoever receives them (meaningful); they truthfully describe how the output was produced (explanation accuracy); and the system operates only under the conditions it was designed for or when it has sufficient confidence (knowledge limits). The second principle is the one design touches most: a correct but unreadable explanation does not meet it.

Why does it matter to a designer?

Because explanation competes with simplicity. Showing a model’s full reasoning can overwhelm people without helping them. Showing nothing leaves them with no tools to judge. The design has to find the right amount for each situation.

A systematic review of explanation interfaces (Cappuccio et al., published on arXiv in 2025 and in ACM Computing Surveys) analyzed 146 papers and concluded that the presentation of explanations is often left as a secondary aspect of system design. Its cross-cutting principles include user-centred design (clear language, lower cognitive load), interactivity and control, multiple levels of detail, and adapting depth to each person’s expertise. Among the gaps, it notes that user-centred design is relatively underexplored in this field and that transparency was explicitly assessed in only eight studies.

A core argument in PAIR’s guidance: because AI products are based on statistics and probability, users should not trust them completely. Explanations help them know when to lean on the system and when to use their own judgment.

How much should you explain?

Four practical criteria:

  1. Match the risk. A rare or high-impact decision needs more explanation than a routine one. A mail filter does not need what a loan decision needs.
  2. Tie it to the action. Show the link between what the user did and the result. “We suggest this plan because you changed your budget” is more useful than a paragraph about the model.
  3. Partial by default. Focus on what affects trust and the decision. A full, replicable explanation makes sense where accountability is required, such as legal or public-sector decisions.
  4. Skip it when it adds nothing. If the result matches what the person expects, or if explaining would expose private data or proprietary methods, there is no need.

How do you present confidence?

A percentage seems the most precise option, but it is not always the most useful. PAIR’s guidance recommends otherwise:

  • Show confidence only if it affects a decision and user testing confirms it helps.
  • Avoid metrics that invite blind acceptance. PAIR warns that a confidence indicator can lead to over-trust, for example when it shows a falsely high score.
  • Prefer categorical levels with a clear action. High, medium and low work if each level says what to do: accept, review, ask for help.
  • In doubtful cases, show alternatives. Offering two or three options invites the user to apply their own judgment.
  • A number needs context. PAIR notes that numeric indicators assume the user has a good grasp of probability. “62%” says little if the person doesn’t know whether that is good for this case.

For how to signal doubt in text, see showing uncertainty.

What to do when reasoning cannot be explained?

With some models, a complete explanation of the reasoning is not possible. PAIR’s guidance proposes alternatives and says the best explanation is likely a partial one:

  • Partial explanations, about the system in general or about one specific output.
  • Example-based explanations, such as similar training cases or cases the system often gets right or wrong.
  • Counterfactuals: explaining why another outcome did not happen (in a hypothetical credit application: “with a 24-month term instead of 36, the payment would have fit your limit”).
  • “What if” exploration, so users can learn the system’s limits.
  • Progressive detail, so curious users can dig deeper without overwhelming everyone else.

When NOT to explain

  • When the explanation is noise. If every answer carries three paragraphs of justification, people stop reading them.
  • When the explanation is false. A plausible justification that does not match the actual calculation is worse than none, and it breaks NIST’s explanation accuracy principle.
  • When it exposes sensitive data. An explanation must not leak information about other users or third parties.
  • When the person only needs the result. An appointment reminder does not need to explain how the time was calculated.

If the explanation concerns using personal data for training, see training data consent.

Example

Illustrative example: a banking app that suggests cutting a subscription cost. The card says: “We suggest cancelling a subscription because it was charged three times in the last 90 days without you using it. This is based on your transactions for that period.” Below, two buttons: “View transactions” and “Don’t suggest this again”. If the person asks “Why not the other one?”, the app replies that subscription was used in the last few days. The explanation is short, tied to the action, says which data it comes from and offers a way to correct it.

Common mistakes

  • Explaining the model’s architecture instead of the reason for the result.
  • Showing a confidence percentage with no context or action.
  • Writing explanations that do not match the real calculation.
  • Hiding which data a recommendation comes from.
  • Testing explanations only with technical teams and not with the people who will read them. See trust calibration for why this matters.

Sources

Frequently asked questions

What is explainability in an AI product?
It is the information the system shows users to understand why it produced a result, what data it relies on and how reliable it is, in language they can use.
Should every AI answer be explained?
No. The amount of explanation should depend on risk and on the decision. A partial explanation is usually better than a complete, unreadable one, and routine tasks may need none.
Should I show a confidence percentage?
Only if it affects a decision and testing shows it helps. A percentage without context can encourage blind acceptance. Categorical levels with a clear action usually work better.
What if the model cannot explain its reasoning?
Offer partial explanations, similar examples, counterfactuals (‘what would have changed the result’) or ‘what if’ exploration, with progressive detail.