RAG (Retrieval-Augmented Generation): Answers Grounded in Your Documents

What is RAG?

RAG combines two parts. A retriever searches your document base for the fragments most related to the question, usually by comparing embeddings. A generator, the LLM, writes the answer using the question and those fragments as context.

The process has three steps: retrieve, augment the prompt with what was retrieved, and generate. If the search brings the right texts, the model has what it needs at hand. If it brings irrelevant texts, the model answers from what it has, and it may invent.

The idea was described in 2020 by Lewis and colleagues (Facebook AI Research, University College London and NYU), who combined a pre-trained generative model (its parametric memory) with a dense index of Wikipedia passages (its non-parametric memory) reached through a neural retriever. They reported better results on three open-domain question answering tasks than models that relied only on parametric memory. The paper also tested two variants: one that uses the same retrieved passages for the whole answer and one that can use different passages for each generated token.

How does it work step by step?

A RAG system has two moments, and it helps to tell them apart because they fail in different ways.

Before anyone asks (indexing):

1. Documents are gathered: guides, articles, tickets, manuals.
2. They are split into fragments, or chunks, for example one per section or paragraph.
3. Each chunk is turned into a vector by an embedding model and stored in an index, often a vector database.

When someone asks (query):

1. The question is turned into a vector with the same model.
2. The system looks for the closest chunks, which resembles a semantic search. Many systems combine this with keyword search (such as BM25), because proper names, codes and acronyms are easier to find by exact match.
3. Optionally, a reranking step reorders the candidates so the most useful ones come first.
4. The best chunks are pasted into the prompt together with the question and the instructions.
5. The model generates the answer, ideally citing which chunk each fact came from.

What matters for design: every step is a place where something can go wrong and where the interface can be honest with the person.

Why does this matter to a designer?

Because RAG defines what your product can answer and how it proves it:

• Visible sources. Since the answer rests on concrete fragments, you can show where each claim comes from. That makes it easier to verify. More in citations and sources in AI answers.
• Explicit limits. If the search finds nothing, the product must say so. “I couldn’t find this in your documents” is a valid and necessary answer.
• Updates without retraining. Changing a document changes the answer. That is useful for living content such as a style guide or a catalog. To change the model’s tone or behavior, compare it with fine-tuning and prompting.
• Latency and cost. Searching and then generating adds steps. Users feel this; show a search state that is different from the writing state.

When NOT to use RAG?

When the task does not depend on documents: writing a creative text, translating a sentence or reasoning about a general problem. Also when the documents are so few that they fit whole in the context window. Anthropic says it plainly in its contextual retrieval post: if your knowledge base is smaller than 200,000 tokens (about 500 pages), you can include all of it in the prompt with no need for RAG, and prompt caching makes that option cheaper and faster. And when the information must be exact and structured, such as balances or inventory, it is better to query the database directly.

What can fail?

Many RAG failures start in retrieval:

• Badly cut chunks. A fragment that cuts a sentence or separates a heading from its content loses meaning.
• Missing context in the fragment. A paragraph that says “this does not apply on mobile” does not say what “this” refers to. Anthropic describes contextual retrieval, which adds a short explanation of each chunk’s context before indexing it. According to its publication, combining contextual embeddings with contextual BM25 reduced the failure rate for the top 20 retrieved chunks by 49 percent (from 5.7% to 2.9%), and adding reranking reduced it by 67 percent in its tests. These are results on its own datasets: treat them as an order of magnitude, not a promise for your case.
• No reranking. Sometimes the right fragment is in the results but not at the top.
• Contradictory documents. The model can mix old and new versions.
• Source manipulation. If anyone can write to the document base, they can influence the answers.
• Answers that drift from the sources. Even with good fragments, the model can add facts that none of them contain. RAG lowers hallucinations; it does not eliminate them.

How do you know it works?

Measure two things separately, because a bad answer can have two different causes:

• Did the right chunk arrive? Build a list of real questions and, for each one, mark which document should show up. Then check whether it appears among the top results. If it does not, the problem is in the search, and no prompt change will fix it.
• Did the model use what it received? With the right chunk in view, check whether the answer stays faithful to it, cites it properly and admits when information is missing.

That list of questions with expected answers is a golden dataset, and the periodic checks are part of what is called evals. Run it again whenever you change the model, the way you cut the documents or the content of the base.

Example

Illustrative example: a design team builds an assistant over its component guide. When a designer asks “what is the card spacing on mobile?”, the system retrieves two fragments: one from the current guide and one from a 2022 version. The answer cites the current one, but the designer also sees the old source and understands why there are two figures. The product shows date and version next to each citation. That interface detail keeps a correct answer from losing credibility.

Common mistakes

• Launching RAG without evaluating retrieval on its own.
• Indexing documents without dates or versions.
• Showing answers without sources because it “looks cleaner”.
• Expecting RAG to replace good document design.
• Assuming more fragments always improve the answer. More noise can make it worse.
• Using RAG out of habit when all the material fits in the context window.

Sources

• Retrieval-augmented generation, Wikipedia
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv:2005.11401)
• Introducing Contextual Retrieval, Anthropic