RAG vs Fine-Tuning: Which One Should You Use?

RAG (retrieval-augmented generation) and fine-tuning are the two main ways to make a general language model useful for your own task. The short version: RAG gives the model knowledge; fine-tuning changes its behaviour. RAG looks up the right information at question time and puts it in the prompt. Fine-tuning trains the model further on your examples, so it permanently behaves differently. Most teams should try good prompts and RAG first, and fine-tune only when there is a clear behaviour problem that prompting cannot fix.

What each one does

RAG leaves the model unchanged. When a question arrives, your application searches a knowledge source (documents, a database, a help centre) for relevant passages, adds them to the prompt, and asks the model to answer from them, ideally with citations. Change the documents and the answers change immediately.

Fine-tuning changes the model’s weights. You collect examples of inputs and the outputs you want, and continue training the model on them, often with a cheap method such as LoRA. The model then follows that style, format or task without being told each time. Its knowledge only changes as much as the examples teach it, and updating it means training again.

Side by side

	RAG	Fine-tuning
Changes	What the model sees for each request	The model itself
Best for	Facts, private documents, frequently changing information, answers with sources	Consistent format or tone, specialised tasks, making a smaller model good at one job
Keeping it current	Update the documents or index; effective immediately	Retrain with new examples
Citations	Natural: you know which passages were used	Not possible from the model alone
Data needed	Your documents	Hundreds to thousands of high-quality labelled examples
Upfront effort	Build ingestion, chunking, search and evaluation	Collect and clean examples, train, evaluate
Cost per request	Higher: longer prompts with retrieved text	Lower prompts; can use a smaller, cheaper model
Main risks	Wrong passages retrieved, too much context	Overfitting, forgetting general skills, stale knowledge
Access control	Filter retrieval by user permissions	Hard: anything learned can be shown to anyone

When to choose RAG

• The answers depend on your documents or data: policies, manuals, product catalogues, tickets.
• The information changes weekly or daily.
• Users need sources they can check.
• Different users may see different data, so permissions matter.

When to choose fine-tuning

• You need a specific output format or style every time, and prompts with examples are not reliable enough or are too long.
• The task is narrow and high-volume, such as classifying tickets or extracting fields, and you want a smaller, cheaper, faster model to do it as well as a large one.
• The model must learn a domain’s way of writing or reasoning that cannot be explained in a prompt.

When to combine them

Many production systems do both: fine-tune a model so it reliably follows the answer format, tone and refusal rules, and use RAG to supply the facts. For example, a support assistant can be fine-tuned on thousands of good past replies (behaviour) and retrieve the current return policy for each question (knowledge).

A practical order of work

1. Prompting: clear instructions, a few examples, a structured output format.
2. RAG: if answers need knowledge the model does not have.
3. Better retrieval and evaluation: hybrid search, reranking, an evaluation set.
4. Fine-tuning: only if a behaviour problem remains, or to cut cost with a smaller model, and only with an evaluation set to prove it helped.

Common mistakes

1. Fine-tuning to add facts. The model may learn some, but it cannot cite them, will not update when they change, and may mix them up. Use retrieval for facts.
2. Blaming the model when retrieval is the problem. Check whether the right passage was retrieved before changing anything else.
3. No evaluation set. Without one, you cannot tell whether either approach helped.
4. Fine-tuning on private data that some users must not see. Anything in training data can come out in answers.

Interview angle

Interviewers ask “RAG or fine-tuning?” to see whether you separate knowledge from behaviour and think about updates, cost and permissions. A strong answer: “RAG for knowledge, fine-tuning for behaviour; start with prompting and RAG, measure, and fine-tune only for a remaining behaviour gap or to move to a cheaper model.” More questions like this are in the LLM interview questions and AI engineer interview questions.