RAG vs CAG: Retrieval vs Cache-Augmented Generation

RAG (retrieval-augmented generation) searches a knowledge base for each question and gives the model only the few chunks that seem relevant. CAG (cache-augmented generation) skips the search: it puts the entire knowledge base into the model’s context once, saves the model’s processed version of it (a cache), and reuses that cache for every question.

CAG became practical because context windows grew to hundreds of thousands of tokens and providers added prompt caching, which makes a repeated long prefix much cheaper and faster on later requests. The name comes from a 2024 paper with the provocative title “Don’t Do RAG”.

How each works

RAG

1. Split documents into chunks, embed them and build a search index.
2. For each question, search the index and take the top few chunks.
3. Put those chunks and the question in the prompt; the model answers.

CAG

1. Put all the documents in one long prompt prefix.
2. Run it through the model once and keep the cache of that prefix (the key-value cache, or the provider’s prompt cache).
3. For each question, append it after the cached prefix; the model answers with all the documents in view.

Side by side

	RAG	CAG
Knowledge base size	Any size, up to millions of documents	Must fit in the context window, with room left for the question and answer
Retrieval errors	Possible: the right chunk may not be found	None: everything is in context
Moving parts	Chunking, embeddings, index, reranking	A long prompt and a cache
Per-question cost	Small prompt, plus search	Long prompt, made cheaper by caching
Latency	Search plus a short prompt	No search; first request slow, later ones faster with a warm cache
Updating knowledge	Re-index changed documents	Rebuild the cache whenever anything changes
Access control	Filter chunks per user	Hard: everyone shares the same cached content, unless you keep one cache per access group
Citations	Natural: you know which chunks were used	Possible, but the model must point to places in a long context

When CAG fits

CAG is a good choice when all of these hold:

• The knowledge base is small and stable: a product manual, an HR policy set, the rules of a game, one contract.
• It fits comfortably in the context window.
• Every user may see all of it.
• Many questions are asked against the same content, so the cache is reused.

RAG remains the right choice for large or fast-changing collections, for content with per-user permissions, and when you need clear citations to specific passages. Long contexts also have a known weakness: models can miss details placed in the middle of a very long prompt (see context windows and hallucinations), so test answer quality before relying on CAG for important answers.

Combining them

The approaches mix well. Some systems cache a stable core (policies, product facts) and use retrieval for the large or changing rest. Others use retrieval to pick a few whole documents rather than small chunks, then rely on a long context to read them fully.

Common mistakes

1. Using CAG for a collection that changes daily, then paying to rebuild the cache constantly.
2. Forgetting permissions. A shared cache shows every user everything in it.
3. Assuming a long context means perfect recall. Measure it on your own questions.
4. Comparing costs without caching. Without prompt caching, sending the whole knowledge base with every question is slow and expensive.

Interview angle

For “RAG vs CAG”:

• One sentence each: retrieve the relevant chunks per question vs preload everything once and cache it.
• When CAG wins: small, stable, shared knowledge that fits in context.
• When RAG wins: large or changing data, per-user access, citations.
• Mention long-context recall limits and the cost role of prompt caching.

Practise with the LLM interview questions.

Key takeaway

RAG finds the right pieces of a large knowledge base for each question. CAG loads a small, stable knowledge base into the context once and reuses the cache, removing retrieval errors and moving parts. Choose CAG for small shared content that fits in context, RAG for everything bigger, changing or permissioned, and combine them when both kinds of content exist.