How to Evaluate a RAG System: Retrieval and Answer Quality

A RAG system can look impressive in a demo and still fail on real questions. Evaluation tells you how good it really is, and which part to fix. The key idea is to measure retrieval and answers separately: if the right passage was never retrieved, no prompt or model change will fix the answer.

Step 1: build a test set

Start with 30 to 100 real questions, taken from search logs, support tickets or the people who will use the system. For each, record:

• the chunk or document that answers it (for retrieval metrics),
• the expected answer or the key facts it must contain (for answer metrics),
• some questions whose answer is not in your documents, to check the system says it does not know.

Keep the test set in version control and add every real failure you find.

Step 2: measure retrieval

Two simple metrics cover most needs:

• Recall@k: the share of questions where the correct chunk appears in the top k results. If it is low, the model rarely sees the answer.
• MRR (mean reciprocal rank): the average of 1 / position of the correct chunk (0 if missing). It rewards putting the right chunk first.

This example evaluates a simple retriever on six labelled questions:

import math, re
from collections import Counter

CHUNKS = {
    "ret-1": "You can return unused items within 30 days of delivery.",
    "ret-2": "Refunds are paid within 14 days of the item reaching our warehouse.",
    "shp-1": "Standard delivery takes 3 to 5 working days.",
    "shp-2": "Express delivery takes 1 working day and costs 9 dollars.",
    "acc-1": "To reset your password, click Forgot password on the sign-in page.",
    "acc-2": "You can change your email address in Account settings.",
}
tok = lambda t: re.findall(r"[a-z0-9]+", t.lower())
df = Counter(w for c in CHUNKS.values() for w in set(tok(c)))
def vec(t):
    tf = Counter(tok(t)); v = {w: tf[w] * (math.log(len(CHUNKS) / df[w]) + 1) for w in tf if w in df}
    n = math.sqrt(sum(x * x for x in v.values())) or 1
    return {w: x / n for w, x in v.items()}
VECS = {cid: vec(c) for cid, c in CHUNKS.items()}

def retrieve(question, k):
    q = vec(question)
    scores = {cid: sum(q.get(w, 0) * x for w, x in v.items()) for cid, v in VECS.items()}
    return [cid for cid in sorted(scores, key=scores.get, reverse=True) if scores[cid] > 0][:k]

# A small labelled test set: each question with the chunk that answers it
TESTS = [
    ("How many days do I have to return something?", "ret-1"),
    ("When will I get my refund?",                   "ret-2"),
    ("How fast is express delivery?",                "shp-2"),
    ("I forgot my password",                         "acc-1"),
    ("How do I update my email?",                    "acc-2"),
    ("Can I send back an item after a month?",       "ret-1"),
]

def evaluate(k):
    hits, rr = 0, 0.0
    for question, answer_id in TESTS:
        results = retrieve(question, k)
        if answer_id in results:
            hits += 1
            rr += 1 / (results.index(answer_id) + 1)
        else:
            print(f"  miss (k={k}): {question!r} -> {results}")
    return hits / len(TESTS), rr / len(TESTS)

for k in (1, 3):
    recall, mrr = evaluate(k)
    print(f"k={k}: recall@{k} = {recall:.2f}, MRR = {mrr:.2f}")

  miss (k=1): 'When will I get my refund?' -> []
  miss (k=1): 'Can I send back an item after a month?' -> ['ret-2']
k=1: recall@1 = 0.67, MRR = 0.67
  miss (k=3): 'When will I get my refund?' -> []
k=3: recall@3 = 0.83, MRR = 0.75

The misses are the useful part:

• “When will I get my refund?” finds nothing, because the chunk says “Refunds” and this simple word-matching retriever treats “refund” and “refunds” as different words. Embeddings, or reducing words to their stems, would fix this.
• “Can I send back an item after a month?” uses different words (“send back”, “a month”) from the chunk (“return”, “30 days”). Keyword matching cannot connect them; semantic search can. See hybrid search.

Each miss points to a specific improvement, and the metrics show whether the improvement worked.

Step 3: measure answers

With retrieval checked, evaluate the generated answers:

• Correctness: does the answer contain the expected facts?
• Faithfulness (groundedness): is every claim supported by the retrieved passages? This catches hallucinations. See context windows and hallucinations.
• Citation accuracy: do the citations point to passages that support the claims?
• Correct refusals: for questions with no answer in the documents, does the system say so?

Check a sample by hand, and use a second model as a judge with a clear rubric for larger sets, comparing its scores with human ratings from time to time.

Step 4: use it every time

Run the evaluation after every change to chunking, embeddings, retrieval settings, prompts or models, and keep a history of the scores. In production, add user feedback and a regular review of failed questions, and turn new failures into test cases.

Common mistakes

1. Only judging final answers, so retrieval problems look like model problems.
2. Test questions written by the builders, which match the documents’ wording too well. Use real user questions.
3. No unanswerable questions, so you never measure made-up answers.
4. Evaluating once and never again after changes.

Key takeaway

Evaluate RAG in two layers: retrieval with recall@k and MRR on labelled questions, then answers for correctness, faithfulness, citations and correct refusals. Study the misses, because each one points to a fix, and rerun the test set after every change. Next: how vector search scales, in vector databases and indexes.