Retrieval-Augmented Generation (RAG) Explained
RAG gives a language model relevant documents at question time by searching a knowledge base first. Learn how it works, why it helps, and where it fails.
Retrieval-augmented generation, or RAG, is a technique in which a system looks up relevant information from an external knowledge source and hands it to a large language model together with the user's question, so the answer is grounded in that material rather than in the model's memory alone. The term comes from a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research, University College London, and New York University. RAG is now the standard way to make a model answer questions about private, specialised, or recent documents without retraining it.
Why does RAG exist?
Language models have built-in limits. Their knowledge stops at the training cutoff. They have never seen your company wiki, contracts, or support tickets. They can hallucinate, producing fluent text that is wrong, and they cannot point to a source. Retraining or fine-tuning a model every time a document changes is slow and expensive, and fine-tuning is a poor way to teach reliable facts. RAG moves the knowledge out of the model's weights and into a searchable store that can be updated in minutes.
How a RAG pipeline works
There are two phases. In the indexing phase, done ahead of time:
- Collect the documents.
- Split them into chunks, typically a few hundred tokens each, often with some overlap.
- Convert each chunk into an embedding, a vector of numbers produced by an embedding model, so that text with similar meaning lands close together in vector space. Word embeddings are the ancestor of this idea.
- Store the vectors, with the original text and metadata, in an index that supports fast approximate nearest-neighbour search.
In the query phase, done for every question:
- Embed the question with the same model.
- Retrieve the top few chunks whose vectors are most similar, usually measured with cosine similarity.
- Build a prompt containing instructions, the retrieved chunks, and the question, using the techniques of prompt engineering.
- Let the model, a transformer, write the answer, ideally citing which chunks it used.
How is retrieval quality improved?
Most RAG failures are retrieval failures: if the right passage never reaches the model, the model cannot use it. Common fixes include:
- Hybrid search, combining classic keyword scoring such as BM25 with vector search, because keywords catch exact names, codes, and rare terms that embeddings blur.
- Re-ranking, where a slower, more accurate model reorders the top fifty or so candidates before the best few are sent to the language model.
- Smarter chunking along headings and sections, so each chunk carries one complete idea.
- Metadata filters for date, product, language, or the user's access rights.
- Query rewriting, which turns a vague or conversational question into a better search query.
Teams evaluate retrieval separately from generation, for example by checking how often the correct passage appears in the top results for a set of test questions.
RAG vs fine-tuning vs long context
| Approach | Best for | Weakness |
|---|---|---|
| RAG | Fresh, private, or large knowledge with citations | Quality depends on retrieval; more moving parts |
| Fine-tuning | Style, format, and domain behaviour | Poor at adding reliable facts; needs retraining |
| Long context | Small document sets that fit in the window | Cost and delay grow with length; details can be missed |
The approaches combine well: a fine-tuned model can follow an organisation's format while RAG supplies the facts.
Limits and risks
RAG reduces hallucination but does not remove it. A model may ignore the supplied text or trust a wrong chunk. Studies of long prompts, such as the 2023 "lost in the middle" paper, found that models use information at the start and end of a prompt better than information buried in the middle, so stuffing in more chunks is not always better. Retrieved text can also contain instructions that hijack the model, a risk called prompt injection, and the retrieval step must enforce the same access controls as the source documents, or private data may leak into answers (see data privacy and AI).
A library card catalog is a good picture of the idea: finding the right card first makes the book worth opening, and a RAG system is the same trick performed on text.
Tags
large language models natural language processing retrieval-augmented generation vector search word embeddings