Chunking strategies compared
Chunking cuts a long document into smaller pieces so a search system can find and hand over only the part that answers a question, and the cutting method decides whether those pieces still make sense.
- 9 min read
- 3 reading levels
- Published
Read these first
On this page 5
One lesson, three depths. Pick the one that fits you today — you can switch any time.
Beginner — No maths. Plain English.
Chunking cuts a document into small pieces, so a search system can return exactly the piece a question needs.
Think about tearing a newspaper page to line a kitchen shelf. Tear straight through the middle of a photo, and the picture is ruined on both halves. Tear along the page's own fold, and each half is still readable on its own. A document behaves the same way when you cut it up for a computer to search.
Why it exists
A model that answers questions cannot read a 200-page manual for every question. It only holds a limited amount of text at once. Reading all of it, every time, would be slow and expensive.
The fix is to cut the document into chunks ahead of time — small, self-contained pieces of text. A search step then picks the two or three chunks that look most relevant. Only those get handed to the model.
That plan works only if each chunk makes sense on its own. A chunk that starts mid-sentence, or ends mid-idea, confuses the search step and the model both.
How it works
The simplest method cuts every fixed number of characters, with no regard for where sentences end.
Document: "...the fourth increase in a row. Analysts had expected a pause..."
Fixed-size cut, no awareness of sentences:
Chunk 7: "...the four"
Chunk 8: "th increase in a row. Analysts had ex..."
^
the word "fourth" is torn in halfA sentence-aware cut waits for a full stop before closing a chunk:
Chunk 7: "...the fourth increase in a row."
Chunk 8: "Analysts had expected a pause..."Neither method understands meaning. Both only look at punctuation or character counts. The next lesson covers a smarter method that looks at what each sentence is actually about.
Where you have already seen it
- "Chat with your PDF" tools. Upload a textbook and ask a question. The answer usually comes from one or two chunks, not the whole file.
- Company support chatbots. A bot answering from a 60-page policy document reads one chunk of it per reply, not the entire thing.
- Google's AI Overviews. A web page gets broken into pieces before one of them is quoted back to you.
Remember this
- A chunk is a small, self-contained piece of a document, cut ahead of time for search.
- Fixed-size chunking is fast but can cut a sentence, or a word, in half.
- Sentence-aware chunking respects sentence endings, at the cost of chunks that vary in size.
What to learn next
- Semantic chunking — cutting by meaning instead of by character count.
- What is RAG? — the bigger system chunking is built to support.
- Tokenization — the other kind of cutting, done to feed a model at all.
Developer — Code and libraries.
Two functions below: one that cuts every N characters, and one that respects sentence boundaries. Comparing their output on the same paragraph makes the difference concrete.
Everything runs on plain Python. No installs, no model download.
Setup
python --version # 3.9 or newer, no packages neededFixed-size versus sentence-aware chunking
import re
text = (
"The Reserve Bank of India raised the repo rate to 6.5 percent. "
"This was the fourth increase in a row. Analysts had expected a pause, "
"but persistent food inflation changed the committee's decision. "
"The rupee strengthened slightly against the dollar after the announcement."
)
def fixed_size_chunks(text, size):
return [text[i:i + size] for i in range(0, len(text), size)]
def sentence_chunks(text, max_chars):
sentences = re.split(r'(?<=[.!?])\s+', text.strip())
chunks, current = [], ""
for sent in sentences:
if current and len(current) + len(sent) + 1 > max_chars:
chunks.append(current.strip())
current = sent
else:
current = (current + " " + sent).strip()
if current:
chunks.append(current.strip())
return chunks
print("=== fixed-size, 80 characters ===")
for i, c in enumerate(fixed_size_chunks(text, 80)):
print(f"[{i}] {c!r}")
print()
print("=== sentence-aware, target 80 characters ===")
for i, c in enumerate(sentence_chunks(text, 80)):
print(f"[{i}] {c!r}")=== fixed-size, 80 characters === [0] 'The Reserve Bank of India raised the repo rate to 6.5 percent. This was the four' [1] 'th increase in a row. Analysts had expected a pause, but persistent food inflati' [2] "on changed the committee's decision. The rupee strengthened slightly against the" [3] ' dollar after the announcement.' === sentence-aware, target 80 characters === [0] 'The Reserve Bank of India raised the repo rate to 6.5 percent.' [1] 'This was the fourth increase in a row.' [2] "Analysts had expected a pause, but persistent food inflation changed the committee's decision." [3] 'The rupee strengthened slightly against the dollar after the announcement.'
Line by line
fixed_size_chunks slices by raw character count. It has no idea what a word or a sentence is. Chunk 0 ends mid-word, splitting "fourth" into "four" and "th" — visible on the boundary between chunks 0 and 1.
sentence_chunks never splits inside a sentence. It keeps adding whole sentences to the current chunk until the next one would push it over max_chars, then starts a new chunk. Chunk 2 in the output is 97 characters, well past the 80-character target, because a single sentence was longer than the target and the function refuses to cut inside it.
That overshoot is the honest trade-off. Respecting sentence boundaries means giving up a hard size guarantee. A production system usually sets max_chars well below the model's real limit, so an occasional long chunk still fits.
Common mistakes
Picking a chunk size and never revisiting it. 500 characters works for short FAQ answers and badly for legal contracts with long clauses. Test on your actual documents.
Chunking a PDF's raw extracted text without cleaning it first. PDF extraction often inserts stray line breaks mid-sentence. Feed that straight into a sentence splitter and it fragments incorrectly, because . and \n no longer line up the way the regex expects.
Forgetting overlap. With no shared text between consecutive chunks, a fact that happens to sit right on a boundary can be split so that neither chunk contains it whole. Most real systems repeat a small amount of text, 10 to 20 percent of the chunk size, at the start of the next chunk.
Assuming smaller chunks are always safer. Very small chunks retrieve precisely but lose surrounding context — a chunk holding only a bare number, with no label, is useless on its own. Later lessons in this section cover fixes for exactly this problem.
Try it yourself
Change max_chars to 40 in sentence_chunks and re-run it. The third sentence is 65 characters wide on its own, well past 40. Predict what happens before you run it.
The function still refuses to split inside that sentence, so it becomes its own chunk anyway, well above your 40-character target. A max_chars smaller than your longest real sentence cannot be honoured as written. A production splitter needs a fallback rule for that case — often a fixed-size cut, applied only to oversized sentences.
What to learn next
- Semantic chunking — splitting by meaning instead of punctuation alone.
- Chunking by document structure — using headings as natural boundaries.
- Hugging Face — where
AutoTokenizerand real chunking tools live.
Researcher — Mathematics and papers.
The chunking design space
Chunking is a hyperparameter search with two competing costs.
- Chunks too large: retrieval becomes coarse. A chunk covering several unrelated ideas gets retrieved for queries about only one of them, wasting context window and diluting relevance signal.
- Chunks too small: each chunk loses surrounding context. A sentence like "It grew by 6 percent that quarter" is retrievable but uninterpretable once separated from what "it" refers to.
There is no closed-form optimum. It depends on the embedding model's effective context length, the retriever's precision at the sizes tested, and the downstream model's tolerance for noisy context. Gao et al. (2023), Retrieval-Augmented Generation for Large Language Models: A Survey (arXiv:2312.10997), treats chunking as one pre-retrieval optimisation among several. No single chunk size dominates across the tasks it surveys.
Overlap as a variance-reduction technique
Given chunk size s and stride s - o for overlap o, take a fact of length f at a random position in the document. Its chance of landing entirely within one chunk is roughly max(0, 1 - f/(s-o)), rather than split across a boundary. Overlap trades index size — more chunks, more storage, more embedding compute — for a lower split probability. Typical production values set o between 10% and 20% of s. That is an empirical convention, not a value derived from this formula, since real documents do not place facts at random.
Retrieval unit versus generation unit
Modern systems increasingly separate the unit that gets embedded and searched from the unit that gets handed to the model. Small units search precisely; large units preserve context for generation. Later lessons build on this split, called parent-document retrieval. It reflects a broader RAG pattern: decoupling retrieval granularity from generation granularity (Gao et al., 2023, section IV).
Key references
- Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401 — the paper that established the retrieve-then-generate pattern chunking exists to support.
- Gao, Y. et al. (2023). Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv:2312.10997
Current state and open problems
Chunk-size selection remains largely empirical. Teams sweep a handful of sizes against a labelled evaluation set and pick whichever scores best, rather than deriving a size from document properties. Learned, content-adaptive chunking — where a model decides boundaries, not a fixed rule — is an active research area. No such method has replaced rule-based splitting as the production default, as of this writing. The real bottleneck is usually not the chunking rule itself. It is the lack of a labelled evaluation set to compare rules against; see Building a retrieval eval set.
What to learn next
- Semantic chunking — a content-aware alternative to fixed rules.
- Building a retrieval eval set — how to measure whether a chunking choice actually helped.
- Long context or retrieval? — when chunking can be skipped altogether.