RAG in Production: Freshness, Permissions, Citations and Cost

A RAG demo answers a few questions about a fixed set of documents. A production RAG system serves many users, every day, over documents that keep changing, some of which not everyone may see. Most of the extra work is data engineering: keeping the index fresh, correct and secure, then measuring quality and cost over time.

Keep the index fresh

Documents are added, edited and deleted all the time. Re-embedding everything each night is slow and expensive. A better approach is incremental indexing, the same idea as incremental loading in data pipelines:

• Give every chunk a stable ID (document ID plus position or section).
• Store a hash of its text. Only re-embed chunks whose hash changed.
• Delete chunks that no longer exist in the source.
• Make every sync idempotent, so running it twice changes nothing (see idempotency).

Enforce permissions in retrieval

If a user may not read a document, its chunks must never reach the model for that user. Store each chunk’s allowed groups as metadata, copied from the source system, and filter before ranking. Never rely on the prompt to hide information.

This example shows both ideas: an index that re-embeds only what changed, removes deleted chunks, and filters results by the user’s groups:

import hashlib

def chunk_id(doc_id, i):
    return f"{doc_id}#{i}"

def content_hash(text):
    return hashlib.sha256(text.encode()).hexdigest()[:12]

index = {}          # chunk id -> {"hash", "text", "groups"}   (vectors would be stored here too)
embed_calls = 0

def embed(text):    # stand-in for an embedding model call, which costs time and money
    global embed_calls
    embed_calls += 1
    return [len(text)]

def sync(doc_id, chunks, groups):
    """Bring one document's chunks in the index up to date. Only new or changed chunks are embedded."""
    seen = set()
    for i, text in enumerate(chunks):
        cid, h = chunk_id(doc_id, i), content_hash(text)
        seen.add(cid)
        if cid not in index or index[cid]["hash"] != h:
            index[cid] = {"hash": h, "text": text, "groups": groups, "vector": embed(text)}
        else:
            index[cid]["groups"] = groups                    # permissions can change without new text
    for cid in [c for c in index if c.startswith(doc_id + "#") and c not in seen]:
        del index[cid]                                       # chunks removed from the document

def search(user_groups):
    """Filter by permissions BEFORE ranking, so restricted text never reaches the model."""
    return [c["text"] for c in index.values() if c["groups"] & user_groups]

sync("returns", ["Return within 30 days.", "Refunds within 14 days."], {"everyone"})
sync("pricing", ["Partner discount is 22%."], {"sales"})
print("first load, embed calls:", embed_calls)

sync("returns", ["Return within 30 days.", "Refunds within 10 days."], {"everyone"})   # one chunk edited
print("after an edit, embed calls:", embed_calls)

sync("returns", ["Return within 30 days."], {"everyone"})                            # a chunk deleted
print("chunks in index:", sorted(index))
print("customer sees:", search({"everyone"}))
print("sales sees:   ", search({"everyone", "sales"}))

first load, embed calls: 3
after an edit, embed calls: 4
chunks in index: ['pricing#0', 'returns#0']
customer sees: ['Return within 30 days.']
sales sees:    ['Return within 30 days.', 'Partner discount is 22%.']

Editing one chunk cost one embedding call, not three. The deleted chunk disappeared from the index. And the partner discount, visible only to the sales group, never appears in a customer’s results.

Citations and “I don’t know”

• Show sources with every answer: document title, section and a link, and the date it was last updated.
• Return a clear “I couldn’t find this in our documents” when retrieval finds nothing relevant, instead of letting the model guess.
• Offer a path forward, such as contacting support, for questions the documents do not cover.

Citations let users check answers, and they let you trace a wrong answer back to the chunk that caused it.

Monitor quality, speed and cost

Track, per day and per version:

• Retrieval health: share of questions with no good match, and recall on your test set.
• Answer quality: user feedback, faithfulness checks on a sample, rate of “I don’t know”.
• Freshness: age of the newest indexed version of each source, and failed syncs.
• Latency: time for retrieval, reranking and generation separately.
• Cost: embedding calls, model tokens and storage per question.

Review the worst-rated answers every week. Most fixes are in the data: a missing document, an outdated page, a bad chunk.

Control cost

• Re-embed only changed chunks.
• Retrieve fewer, better passages with reranking, so prompts stay short.
• Cache answers for repeated questions, with expiry when sources change.
• Use a smaller model for simple questions and route harder ones to a larger model.

Common mistakes

1. Full re-indexing every night, which is slow, costly and leaves a window of inconsistency.
2. Never deleting, so removed or outdated documents keep appearing in answers.
3. Permissions checked in the prompt instead of in retrieval.
4. No monitoring after launch, so quality drifts as documents change.

Key takeaway

Production RAG is mostly careful data engineering: incremental, idempotent indexing with deletions; permissions enforced in retrieval; citations and honest “I don’t know” answers; and monitoring of quality, freshness, latency and cost. With these in place, the RAG course is complete. Continue with the AI engineer roadmap, or test yourself with the AI engineer interview questions.