Hallucinations: Why AI Makes Things Up and How to Design for It

A hallucination is an AI answer that sounds confident but is false. Why it happens, how to reduce it and how to design so that people do not trust it blindly.

info Quick Definition
A hallucination in AI is a response generated by a model that contains false or misleading information presented as fact. It is an effect of how text is generated, not an intentional lie, and it often sounds as confident as a correct answer.

What is a hallucination?

An LLM generates text one token at a time: it computes how likely each continuation is and picks one. That process produces correct-sounding, convincing sentences even when the content is false. A made-up citation, a wrong date, a software function that does not exist or a statistic with no origin are typical examples.

A reference academic survey (Ji et al., published in ACM Computing Surveys) distinguishes two types. An intrinsic hallucination is an output that contradicts the source content, for example a summary that says the opposite of the document. An extrinsic one is an output that cannot be verified from that source: it is neither backed up nor refuted by it. For people using the product the difference matters little: in both cases the answer looks true.

It is also called confabulation. Some critics object to the word “hallucination” because it humanizes a program that perceives nothing; this wiki uses it because it is the term most people already know.

Why does it happen?

There are several causes, often combined:

  • The training objective. The model learns to predict plausible text. If the exact fact is not in what it learned, it can still produce something that sounds right.
  • Evaluations that reward guessing. A paper by OpenAI researchers (Kalai, Nachum, Vempala and Zhang, 2025) argues that current training and evaluation reward guessing over admitting uncertainty: on an exam, a random answer can earn points and “I don’t know” earns none. They propose changing how benchmarks are scored.
  • Imperfect data. Training data can be incomplete, outdated or repeat common mistakes from the internet.
  • Missing context. If the relevant information is not in the context window, the model fills the gap with what seems likely.
  • Questions beyond its reach. Asking for very recent or company-internal details without giving it access to sources raises the risk.

Temperature changes how much the answer varies, but lowering it does not fix a fact the model does not have.

How do you reduce them?

No method eliminates them. Anthropic’s documentation puts it this way: these techniques reduce hallucinations but do not eliminate them, and critical information must be validated. The ones it recommends, useful in any product:

  • Allow “I don’t know”. Explicitly giving the model permission to admit it lacks enough information reduces invented answers.
  • Ask for direct quotes first. With long documents, asking the model to extract the exact sentences before answering anchors the response to the real text. If it cannot find a quote that supports a claim, it must withdraw the claim.
  • Restrict it to the documents provided. Tell it to use only the supplied information, not its general knowledge. This is the idea behind RAG.
  • Compare several runs. If the same question gets different answers, something is doubtful.
  • Evaluate with real cases. A set of questions with known answers lets you measure whether the system gets worse after a change. See evals.

Do not assume RAG solves it. A Stanford study (Magesh and others, published in the Journal of Empirical Legal Studies in 2025) evaluated legal research tools from LexisNexis and Thomson Reuters, which advertise retrieval of real documents. They hallucinated less than a general chatbot such as GPT-4, but still more than 17% of the time.

What happens when nobody checks?

Two public cases show the cost:

  • Mata v. Avianca (2023, New York). A lawyer filed six cited cases that ChatGPT had invented. Judge P. Kevin Castel fined the lawyers $5,000 for bad faith.
  • Moffatt v. Air Canada (2024, British Columbia). The airline’s chatbot told a passenger he could request the bereavement fare after buying the ticket, which the policy did not allow. The tribunal held the airline responsible for what its own chatbot said and ordered it to pay.

The design lesson is the same in both: responsibility does not move to the model. If your product shows an answer, your product answers for it.

How do you design for them?

Design cannot promise accuracy, so it should make verification easy. Microsoft’s HAX guidelines give a base: make clear how well the system does what it does (G2), support efficient correction (G9), scope services when in doubt (G10) and make clear why the system did what it did (G11). In practice:

  • Show the source next to the claim, not only at the end of the answer. See citations and sources.
  • Visually separate what came from a document from what the model generated.
  • On high-stakes topics (health, money, legal, safety), do not present the answer as final; offer human review or a verifiable next step.
  • Provide a simple way to correct (“this is not right”), with feedback the team can use. See user feedback.
  • When the system has no basis to answer, have it say “I did not find this in your documents” instead of improvising. More in how to show uncertainty and trust calibration.

Example

Illustrative example: an accounting app helps classify expenses and, in one case, suggests a tax deduction that does not exist in the person’s country. The team cannot fully prevent this, but it can design around it: show the rule it applied with its source, ask for confirmation before saving, and flag suggestions that depend on local rules. The person sees where the suggestion comes from and can reject it without losing their work.

Common mistakes

  • Believing a newer model no longer hallucinates. It lowers the frequency; it does not eliminate it.
  • Believing RAG removes the problem.
  • Showing answers in the same tone for trivial and critical topics.
  • Trusting generated citations without opening the source.
  • Not measuring the error rate before launch.
  • Hiding sources because they “clutter” the interface.

Sources

Frequently asked questions

What is a hallucination in AI?
It is a response generated by a model that contains false or misleading information presented as fact. It usually sounds plausible and well written.
Why do models hallucinate?
Because they learn to predict likely text, not to verify facts, and because many evaluations reward guessing over admitting that something is not known. Incomplete or erroneous data and missing context also contribute.
Can hallucinations be eliminated?
Not completely. They can be reduced with good context, source retrieval, instructions that allow “I don’t know” and evaluation, but even Anthropic’s documentation states that these techniques do not eliminate them. The design should assume they still happen.
How do you design an interface for hallucinations?
Show verifiable sources next to each claim, let people correct the answer and avoid presenting it as final on high-stakes topics.
Does a system with RAG no longer hallucinate?
It hallucinates less, but it does not stop. A Stanford study of AI legal research tools that retrieve documents found hallucinations in more than 17% of answers.