Chunking Strategies for RAG: Fixed, Semantic, and Recursive

Chunking is the most underrated lever in RAG. The same documents, embedding model, and retriever produce wildly different retrieval quality depending on how you split the text. Chunks that are too large dilute meaning; chunks that are too small lose context. This guide covers the four main strategies, when each wins, and how to measure the difference.

Why Chunking Matters

The retriever matches queries against chunks, not documents. If the answer to a query spans a chunk boundary, retrieval fails even with a perfect embedding model. If a chunk contains five unrelated topics, its embedding is a blurry average that matches nothing well.

The trade-off is fundamental:

• Small chunks: precise embeddings, but answers get cut off and context is lost.
• Large chunks: full context, but embeddings average away the specific meaning.

Strategy 1: Fixed-Size Chunking

Split by character or token count with overlap. The simplest strategy and a fine baseline.

def fixed_chunks(text: str, chunk_size: int = 800, overlap: int = 100) -> list[str]:
    chunks = []
    start = 0
    while start < len(text):
        end = min(start + chunk_size, len(text))
        chunks.append(text[start:end])
        start = end - overlap
    return chunks

Pros: trivial, deterministic, predictable token counts. Cons: cuts sentences and paragraphs in half, ignores document structure. Use it as the baseline you compare everything else against.

Strategy 2: Recursive Character Chunking

Split on a hierarchy of separators: paragraphs first, then sentences, then words. This respects natural boundaries instead of cutting mid-sentence.

SEPARATORS = ["\n\n", "\n", ". ", " ", ""]

def recursive_chunks(text: str, chunk_size: int = 800, overlap: int = 100) -> list[str]:
    chunks = []

    def split(text: str, sep_index: int) -> list[str]:
        if len(text) <= chunk_size or sep_index >= len(SEPARATORS):
            return [text]
        sep = SEPARATORS[sep_index]
        parts = text.split(sep)
        result = []
        current = ""
        for part in parts:
            candidate = current + sep + part if current else part
            if len(candidate) > chunk_size and current:
                result.append(current)
                current = part
            else:
                current = candidate
        if current:
            result.append(current)
        # Recurse into oversized pieces with the next separator
        return [piece for p in result for piece in split(p, sep_index + 1)]

    return split(text, 0)

This is the default in LangChain's RecursiveCharacterTextSplitter and is the right starting point for most documents: it keeps paragraphs and sentences intact while bounding chunk size.

Strategy 3: Semantic Chunking

Split where the meaning changes. Embed sentences, compare adjacent similarity, and break when the similarity drops below a threshold.

import numpy as np

def semantic_chunks(sentences: list[str], embed, threshold: float = 0.75) -> list[str]:
    """Split sentences into chunks where semantic similarity drops."""
    if not sentences:
        return []
    embeddings = np.array([embed(s) for s in sentences])
    chunks = [[sentences[0]]]
    for i in range(1, len(sentences)):
        sim = cosine(embeddings[i], embeddings[i - 1])
        if sim < threshold:
            chunks.append([])
        chunks[-1].append(sentences[i])
    return [" ".join(c) for c in chunks]

Pros: chunks align with topics, which is exactly what semantic retrieval wants. Cons: expensive (one embedding per sentence), threshold tuning required, and it can produce wildly uneven chunk sizes. Use it for conversational or narrative content where topic shifts are the real boundaries.

Strategy 4: Document-Aware Chunking

Use the document's own structure: markdown headings, HTML tags, code blocks, PDF sections. Each structural unit becomes a chunk, with parent-child relationships preserved.

import re

def markdown_chunks(markdown: str) -> list[dict]:
    """Chunk markdown by headings, keeping the heading path as context."""
    chunks = []
    current_heading = ""
    current_text = []

    for line in markdown.splitlines():
        heading = re.match(r"^(#{1,6})\s+(.*)", line)
        if heading:
            if current_text:
                chunks.append({
                    "heading": current_heading,
                    "text": "\n".join(current_text),
                })
            current_heading = line
            current_text = []
        else:
            current_text.append(line)

    if current_text:
        chunks.append({"heading": current_heading, "text": "\n".join(current_text)})
    return chunks

Store the heading path in the chunk's metadata so the LLM can cite "Section 3.2: Refund Policy" instead of "chunk 47". This is the strongest strategy for structured documents: docs, wikis, READMEs, and anything with headings.

Chunk Size: The Numbers

Chunk Size (tokens)	Retrieval Precision	Context Completeness	Cost
100-200	High	Low	Low
300-500	Balanced	Balanced	Mid
800-1200	Low	High	High

Start at 300-500 tokens with 10-20% overlap. Tune from there based on your measured retrieval quality, not intuition.

Evaluating Chunk Quality

Chunking changes are measurable. Build a labeled set of queries with the documents that should be retrieved, then compare strategies:

def evaluate_chunking(chunker, documents, queries, relevant_docs, k=5):
    index = build_index(chunker, documents)
    total_recall = 0.0
    for query, relevant in zip(queries, relevant_docs):
        results = index.search(query, k)
        hit = set(results) & set(relevant)
        total_recall += len(hit) / len(relevant)
    return total_recall / len(queries)

Run every strategy on the same data and pick the winner. Also check answer quality: retrieval metrics do not capture whether the LLM could actually answer from the chunk. Spot-check generated answers for completeness.

Practical Rules

1. Match the chunker to the document type: markdown-aware for docs, recursive for prose, semantic for conversations.
2. Keep the heading path in metadata for citation and context.
3. Overlap 10-20% to avoid cutting answers at boundaries.
4. Never chunk code by characters: split on function/class boundaries.
5. Re-evaluate after any change: chunk size, model, or strategy changes all shift retrieval quality.
6. Consider parent-child retrieval: retrieve small chunks, then feed the parent section to the LLM for context.

Implementation Checklist

• Start with recursive chunking at 300-500 tokens, 10-20% overlap
• Switch to document-aware chunking for structured content
• Use semantic chunking only for topic-shifting narrative content
• Store heading paths and source metadata in every chunk
• Build a labeled query set and compare strategies with recall@k
• Spot-check answer completeness, not just retrieval metrics
• Re-evaluate after every chunking or model change

---

MatterAI builds frontier AI infrastructure for engineering teams — from inference-optimized models to autonomous coding agents and agentic code reviews.

Explore what we're building:

• Orbital IDE — Autonomous AI coding agent with background agents and deep codebase memory
• AI Code Reviews — Agentic pre-commit reviews across GitHub, GitLab, and Bitbucket
• Axon Models — Frontier-grade reasoning models at 70% lower inference cost

Get started free - https://app.matterai.so

• Read the docs
• View pricing

---

Follow us on X · LinkedIn · GitHub