Chunking Strategies for RAG: How to Split Documents

Retrieval works on chunks, not whole documents. A chunk is a piece of a document small enough that its embedding captures one clear topic, and short enough to fit several into a prompt. How you split documents has a big effect on answer quality: a chunk that cuts a sentence in half, or loses the heading that explains what it is about, can make the right answer impossible to find.

Why not embed whole documents?

A long document covers many topics. Its embedding becomes an average of all of them, so it matches no single question well. And even if it is retrieved, the whole document may not fit in the context window, or it adds a lot of irrelevant text. Smaller chunks make retrieval precise and prompts focused.

But chunks that are too small lose context. “The limit is 30 days” means little without knowing it is about returns.

Two common approaches

Fixed-size chunks split text every N tokens or words, with some overlap so that a sentence cut at a boundary appears whole in at least one chunk. It is simple and works on any text.

Structure-based chunks split on the document’s own structure: headings, sections, paragraphs or list items. Chunks follow natural topic boundaries. Adding the title and heading path to each chunk keeps its context.

This example splits the same policy both ways:

import re

DOC = """# Returns policy
## Time limit
You can return unused items within 30 days of delivery.
## Refunds
Refunds are paid within 14 days of the item reaching our warehouse. Shipping costs are not refunded.
## Exceptions
Gift cards and personalised items cannot be returned."""

def fixed_size(text, size, overlap):
    """Split into windows of `size` words, each starting `size - overlap` words after the previous one."""
    words = re.sub(r"#+ ", "", text).split()
    step = size - overlap
    return [" ".join(words[i:i + size]) for i in range(0, max(len(words) - overlap, 1), step)]

def by_heading(text):
    """Split on second-level headings and prefix each chunk with its title path."""
    title = re.search(r"^# (.+)$", text, re.M).group(1)
    chunks = []
    for section in re.split(r"^## ", text, flags=re.M)[1:]:
        heading, body = section.split("\n", 1)
        chunks.append(f"{title} > {heading}: {' '.join(body.split())}")
    return chunks

print("FIXED SIZE (12 words, overlap 3):")
for c in fixed_size(DOC, 12, 3):
    print("  |", c)
print("BY HEADING:")
for c in by_heading(DOC):
    print("  |", c)

FIXED SIZE (12 words, overlap 3):
  | Returns policy Time limit You can return unused items within 30 days
  | within 30 days of delivery. Refunds Refunds are paid within 14 days
  | within 14 days of the item reaching our warehouse. Shipping costs are
  | Shipping costs are not refunded. Exceptions Gift cards and personalised items cannot
  | personalised items cannot be returned.
BY HEADING:
  | Returns policy > Time limit: You can return unused items within 30 days of delivery.
  | Returns policy > Refunds: Refunds are paid within 14 days of the item reaching our warehouse. Shipping costs are not refunded.
  | Returns policy > Exceptions: Gift cards and personalised items cannot be returned.

The fixed-size chunks cut sentences and mix topics: the second chunk ends in the middle of the refund rule. The heading-based chunks each hold one complete topic, and each says which policy and section it comes from. Real chunk sizes are much larger than 12 words; the small size just makes the problem easy to see.

Practical guidance

• Use the document’s structure when it has one: headings in Markdown or HTML, sections in PDFs, functions in code.
• Fall back to fixed size with overlap for long sections or unstructured text. A few hundred tokens per chunk, with 10 to 20% overlap, is a common starting point.
• Prefix chunks with context: document title, section heading, and sometimes a one-line summary.
• Keep metadata with each chunk: source, URL, date, owner and access permissions.
• Keep tables and code blocks whole where possible; splitting them in the middle makes them unreadable.
• Test, don’t guess. Try two or three strategies and sizes, and compare them with a retrieval test set (see evaluating RAG).

Common mistakes

1. One chunk size for everything: FAQs, contracts and code need different treatment.
2. Losing the heading, so chunks lose their meaning.
3. Chunks so small that answers are split across several, none of which is retrieved on its own.
4. Forgetting to re-chunk and re-embed when you change the strategy.

Key takeaway

Chunk documents so each piece holds one clear topic with enough context to stand alone. Prefer splitting on structure, add titles and headings to each chunk, use fixed size with overlap as a fallback, and test sizes against real questions. Next: finding the right chunks with hybrid search and reranking.