Why chunking matters so much
In RAG, the retriever can only return what’s in the index, and the LLM can only answer from what the retriever returns. Each chunk becomes one embedding — one point in vector space summarising everything in it. If a chunk mixes three topics, its vector is a blurry average that matches none of them well. If a chunk is half an answer, the model gets half the facts.
A good chunk is a self-contained unit of meaning — something a human could read on its own and understand what it’s about.
The chunk-size trade-off
| Small chunks (~100–256 tokens) | Large chunks (~1,000+ tokens) | |
|---|---|---|
| Retrieval precision | High — focused vectors match specific questions | Lower — relevant sentence is diluted |
| Context for the LLM | Can be missing definitions, conditions, exceptions | Rich surrounding context |
| Tokens per answer | Fewer per chunk, but you may need more chunks | More — higher cost and latency |
| Index size | More vectors | Fewer vectors |
| Best for | FAQs, definitions, specific facts | Explanations, procedures, narrative reports |
Live chunking playground
6 chunking strategies compared
1. Fixed-size chunking
Split every N characters or tokens, optionally with overlap. Simple, fast and predictable — but blind to meaning, so it routinely cuts sentences, lists and tables in half. Fine for a quick baseline or uniform text like logs.
2. Recursive chunking
Try to split on the biggest natural boundary first (blank lines between paragraphs), and only if a piece is still too big, fall back to smaller separators (newlines, then sentences, then words). This is what LangChain’s popular RecursiveCharacterTextSplitter does, and it’s the most sensible general-purpose default.
from langchain_text_splitters import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=800, # characters (~200 tokens); use a token-based length function for precision
chunk_overlap=100,
separators=["\n\n", "\n", ". ", " ", ""],
)
chunks = splitter.split_text(document_text)3. Document-structure-aware chunking
Use the document’s own structure — Markdown or HTML headings, PDF sections, slide boundaries, code functions — as chunk boundaries. Each chunk maps to a real section, and the heading path (“Leave Policy › Parental leave › Pay”) becomes valuable metadata. Usually the biggest single quality win for well-structured documents.
4. Semantic chunking
Embed each sentence and start a new chunk wherever similarity between neighbouring sentences drops sharply — i.e., where the topic changes. Great for long unstructured text like transcripts. It costs extra embedding calls at indexing time and doesn’t always beat structure-aware splitting, so benchmark it.
5. Parent-child (small-to-big) retrieval
Index small child chunks (e.g., 200 tokens) for precise matching, but when one matches, send the LLM its larger parent section (e.g., 1,000+ tokens). You get sharp retrieval and full context. A close cousin, the sentence-window approach, returns a few sentences either side of the matched sentence.
6. Contextual and late chunking
Both tackle the “orphan chunk” problem — a chunk saying “it increases by 5% annually” with no clue what “it” is. Contextual chunking uses an LLM to prepend a short summary of where the chunk sits in the document before embedding. Late chunking embeds the whole document with a long-context embedding model first, then pools token embeddings per chunk so each vector carries document-wide context.
| Strategy | Quality | Cost / complexity | Use when |
|---|---|---|---|
| Fixed-size | ★☆☆ | Lowest | Baselines, uniform text |
| Recursive | ★★☆ | Low | General-purpose default |
| Structure-aware | ★★★ | Low–medium | Docs with headings: policies, manuals, wikis |
| Semantic | ★★☆–★★★ | Medium | Transcripts, long unstructured prose |
| Parent-child | ★★★ | Medium | Precise questions that need surrounding context |
| Contextual / late | ★★★ | Medium–high | Chunks that lose meaning out of context |
Recommended chunk sizes by content type
Starting points, not rules — always validate on your data and embedding model:
| Content | Starting size | Notes |
|---|---|---|
| FAQs, glossaries | One Q&A or term per chunk | Natural units — don’t merge them |
| Policies, manuals, wikis | ~300–800 tokens per section | Split on headings; keep heading path as metadata |
| Research papers, reports | ~500–1,000 tokens | Respect sections; keep figure captions with text |
| Chat logs, transcripts | ~200–500 tokens or semantic | Keep speaker turns intact |
| Source code | Function / class level | Use language-aware splitters |
| Tables | Whole table (or row groups + header) | Convert to Markdown; never split headers from rows |
Also respect your embedding model’s maximum input length — anything beyond it is silently truncated. Measure real token counts with the Token Counter.
How much overlap?
Overlap repeats the end of one chunk at the start of the next, so a fact straddling a boundary appears whole in at least one chunk. 10–20% of chunk size is a common default. With structure-aware splitting at clean section boundaries you may need little or none; with fixed-size splitting it matters more. Beyond ~25% you mostly pay for duplicate storage, duplicate retrieval results and extra tokens.
Metadata and contextual chunks
Store more than the text. Useful metadata per chunk:
- Source: document title, URL, page, section heading path — for citations.
- Filters: date, version, region, product, department, access group.
- Context: prepend “Document: Leave Policy v7 › Section: Parental leave” to the chunk text before embedding — a cheap, effective form of contextual chunking.
How to pick the winning strategy
- Collect 50–200 real user questions and mark which passage answers each.
- Index the same documents with 2–4 candidate configurations (strategy × size × overlap).
- Measure recall@k (is the right passage in the top k?) and MRR for each.
- For the top two, run end-to-end and score answer faithfulness and correctness.
- Factor in tokens per answer — the cheapest config within a point or two of the best usually wins.
Discussion