What Is RAG? Retrieval-Augmented Generation Explained with Code

Retrieval-augmented generation (RAG) is a way of answering questions with a language model using your own information. Before the model answers, the system retrieves the most relevant passages from your documents and adds them to the prompt. The model then generates an answer from those passages, ideally with citations.

RAG solves three problems at once. Models do not know your private documents. Their knowledge stops at their training date. And they sometimes make things up. With RAG, the knowledge lives in your documents, which you can update any time, and every answer can point to its source.

The two halves of a RAG system

1. Indexing (done in advance, and again when documents change)

• Load documents from their sources.
• Split them into chunks small enough to retrieve precisely. See chunking strategies.
• Embed each chunk into a vector, and store it with its text and metadata (source, date, permissions).

2. Answering (for every question)

• Embed the question with the same model.
• Retrieve the most similar chunks, often with hybrid search and reranking.
• Build a prompt with instructions, the chunks and the question.
• Generate the answer, with citations.

A small RAG pipeline you can run

This example builds the retrieval side of a RAG system over three help-centre documents. It uses TF-IDF vectors, a classic word-weighting method, as a stand-in for an embedding model, so it runs anywhere without downloads. A real system would call an embedding model in step 2; everything else stays the same.

import math, re
from collections import Counter

DOCS = {
    "returns.md":  "Returns. You can return unused items within 30 days of delivery. "
                   "Start a return from the Orders page. Refunds are paid within 14 days of the item reaching our warehouse.",
    "shipping.md": "Shipping. Standard delivery takes 3 to 5 working days. Express delivery takes 1 working day "
                   "and costs 9 dollars. We ship to 40 countries.",
    "account.md":  "Account. To reset your password, click Forgot password on the sign-in page. "
                   "You can change your email address in Account settings.",
}

def tokenize(text):
    return re.findall(r"[a-z0-9]+", text.lower())

# 1. Split documents into chunks (here: one sentence per chunk) and keep the source for citations
chunks = [(name, s) for name, text in DOCS.items() for s in re.split(r"(?<=\.)\s+", text) if len(s.split()) > 2]

# 2. Index: TF-IDF vectors (a simple stand-in for an embedding model)
df = Counter(w for _, c in chunks for w in set(tokenize(c)))
idf = {w: math.log(len(chunks) / n) + 1 for w, n in df.items()}
def vector(text):
    tf = Counter(tokenize(text))
    v = {w: tf[w] * idf.get(w, 0) for w in tf}
    norm = math.sqrt(sum(x * x for x in v.values())) or 1
    return {w: x / norm for w, x in v.items()}
index = [(name, c, vector(c)) for name, c in chunks]

# 3. Retrieve the chunks most similar to the question
def retrieve(question, k=2):
    q = vector(question)
    scored = [(sum(q.get(w, 0) * x for w, x in v.items()), name, c) for name, c, v in index]
    return sorted(scored, reverse=True)[:k]

# 4. Build the prompt the model would receive
question = "How long do refunds take?"
hits = retrieve(question)
context = "\n".join(f"[{i}] ({name}) {c}" for i, (_, name, c) in enumerate(hits, 1))
prompt = ("Answer the question using only the sources below. Cite sources like [1]. "
          "If the answer is not there, say you don't know.\n\n"
          f"Sources:\n{context}\n\nQuestion: {question}")
print(prompt)

Answer the question using only the sources below. Cite sources like [1]. If the answer is not there, say you don't know.

Sources:
[1] (returns.md) Refunds are paid within 14 days of the item reaching our warehouse.
[2] (shipping.md) We ship to 40 countries.

Question: How long do refunds take?

The first source answers the question, so a model would reply something like “Refunds are paid within 14 days of the item reaching the warehouse [1].” Notice the second source: it is not relevant, but it was included because we always take the top two. Real systems handle this with a minimum similarity score, a reranker, or by letting the model ignore unhelpful sources. Retrieval quality is where most of the work in RAG goes.

When to use RAG

• Answers must come from your documents: policies, manuals, product data, tickets.
• Information changes, and answers must stay current.
• Users need sources they can check.
• Different users may see different documents, so you filter retrieval by permissions.

For changing a model’s style or format instead of its knowledge, see RAG vs fine-tuning.

How RAG systems fail

1. The right passage is not retrieved: poor chunking, vocabulary mismatch, or missing documents. This is the most common failure.
2. Too many irrelevant passages distract the model or crowd the context window.
3. The model ignores or misreads the sources, or adds facts that are not in them.
4. Stale or duplicate documents give outdated or conflicting answers.
5. Permissions are ignored, so users see passages they should not.

Each of these can be measured and fixed, which is what the rest of this course covers. See evaluating RAG.

Key takeaway

RAG retrieves relevant passages from your documents and asks the model to answer from them, with citations. It has an indexing half (load, chunk, embed, store) and an answering half (embed the question, retrieve, prompt, generate). Most problems are retrieval problems, so invest there first. Next: chunking strategies.