RAG (retrieval-augmented generation)

RAG, or retrieval-augmented generation, means searching your own documents first and pasting the relevant pieces into the prompt, so the model answers from those instead of from memory.

An open-book exam changes what the student needs. Without the book, they answer from whatever they memorised months ago, and the gaps get filled with guesses. With the book, the same student turns to the right page and answers from the text in front of them. RAG hands the model the book, opened at the right page.

There is no magic in it. The retrieval step is ordinary search — usually embedding search — and the generation step is an ordinary prompt that happens to contain the search results.

The pipeline#

Once, up front:
  Documents → split into chunks → embed each chunk → store in a vector database

Every question:
  Question → embed → find nearest chunks → build prompt with those chunks
           → model answers, citing which chunk it used

Why teams reach for it#

It handles facts that change. A price list updated this morning is available to the model this morning, with no retraining. It gives you citations, so a reader can check the claim against the source paragraph. It keeps private data out of any training process. And it is far cheaper and faster to set up than fine-tuning, which teaches behaviour well and facts badly.

RAG reduces hallucination but does not remove it, and it is worth knowing where it breaks. If retrieval returns the wrong chunks, the model answers confidently from wrong material — retrieval quality sets the ceiling on the whole system. Questions that need the whole corpus at once ("what are the themes across all 400 reports?") do not fit the pattern. And every retrieved chunk eats context window, so more chunks is not automatically better.

Most of the practical work in RAG is unglamorous: how you split documents, how big the chunks are, whether you overlap them, and whether you re-rank the results before they reach the model.

Where to go next#

• Full lesson: What is RAG?
• Related terms: embedding, vector-database, context-window, hallucination