Learn AI Create the Future — with Pranay
Main site

Chunking and Long Documents

Retrieve small, return big

Parent-document retrieval searches over small, precise chunks but hands the model the larger passage each one came from, combining accurate search with enough context to actually answer.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. Where you have already seen it
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Parent-document retrieval searches with small chunks, but returns the larger passage each one belongs to.

Think about a library index card. It lists a book's title and one key sentence about it. Pulling that card sends you to fetch the entire book from the shelf.

Why it exists

Earlier lessons in this section ran into a tension. Small chunks search precisely, since a short, focused piece of text matches a specific question well. Large chunks read well — they keep enough surrounding text for a person, or a model, to follow.

Small chunks alone often leave the model with too little to answer from. Large chunks alone make search blurry, matching a whole paragraph when only one sentence in it was relevant.

Parent-document retrieval keeps both. It searches over small, precise pieces — the "child" chunks — and returns the larger "parent" chunk each one belongs to.

How it works

Document, split into small child chunks for searching:

  Child 1: "The RBI raised the repo rate to 6.5 percent in February."
  Child 2: "The move aimed to control inflation, above target for three quarters."

Both children map back to the same parent paragraph.

Question: "Why did the RBI raise rates?"
  -> best match: Child 2 (precise match, mentions "control inflation")
  -> returned to the model: the FULL parent paragraph, both sentences,
     not Child 2 alone

The search step stays sharp. The model still gets the full picture.

Where you have already seen it

  • Search engine snippets. Google's result snippet is short, but clicking it opens the whole page, not that one sentence alone.
  • E-book search. Searching a Kindle book for a phrase jumps you to the full page it appears on, not an isolated line.
  • Legal research tools. A matched clause opens the whole contract section around it, so the clause is read in context.

Remember this

  • Small chunks are searched. Large chunks are returned. They do not have to be the same size.
  • This solves the "too thin to be useful, too broad to be precise" trade-off in one move.
  • Every small chunk needs a stored link back to its larger parent.

What to learn next

Developer — Code and libraries.

This searches small sentence-level chunks with TF-IDF, then returns the larger paragraph each matched sentence belongs to.

Setup

bash
pip install scikit-learn

Searching small, returning big

parent_doc.py
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

# Two parent paragraphs, each split into small child sentences for precise search.
parents = [
    "The Reserve Bank of India raised the repo rate to 6.5 percent in its "
    "February meeting. The move was aimed at controlling inflation, which had "
    "stayed above the RBI's 4 percent target for three straight quarters.",
    "Infosys reported quarterly revenue of 38,000 crore rupees, a rise of 6 "
    "percent year on year. The company also announced a share buyback worth "
    "9,300 crore rupees, its fourth in six years.",
]

children = []   # (child_text, parent_index)
for pid, para in enumerate(parents):
    for sent in para.split(". "):
        sent = sent.strip().rstrip(".")
        if sent:
            children.append((sent, pid))

child_texts = [c[0] for c in children]
vec = TfidfVectorizer(stop_words="english").fit(child_texts)
child_vecs = vec.transform(child_texts)

def search(query, k=1):
    qv = vec.transform([query])
    sims = cosine_similarity(qv, child_vecs)[0]
    best = sims.argsort()[::-1][:k]
    results = []
    for i in best:
        child_text, parent_id = children[i]
        results.append({
            "matched_sentence": child_text,
            "score": round(float(sims[i]), 3),
            "parent_returned": parents[parent_id],
        })
    return results

for r in search("What was the buyback amount?"):
    print("matched:", r["matched_sentence"])
    print("score:", r["score"])
    print("returned to the model:")
    print(" ", r["parent_returned"])
Output
matched: The company also announced a share buyback worth 9,300 crore rupees, its fourth in six years
score: 0.329
returned to the model:
  Infosys reported quarterly revenue of 38,000 crore rupees, a rise of 6 percent year on year. The company also announced a share buyback worth 9,300 crore rupees, its fourth in six years.

Scores come from plain TF-IDF cosine similarity on a two-document toy corpus. On a real corpus of thousands of chunks, the exact number will differ. What matters is the pattern: a small match, a large return.

Line by line

children stores each sentence alongside its parent_index, the single most important line in the file. Without that mapping, there is no way back to the parent once a child chunk wins the search.

TfidfVectorizer(stop_words="english") matters here. Without stop-word removal, common words like "the" and "was" pollute similarity scores on a tiny corpus like this one, occasionally outranking a genuine keyword match. On a large real corpus this matters less, but it costs nothing to include and avoids a confusing debugging session.

search() finds the best child, then looks up parents[parent_id]. The function returns the parent text, never the child text, in its parent_returned field. That is the entire pattern — search small, return big — in two lines.

Common mistakes

Storing children without a reliable parent link. If chunk IDs get reshuffled during a re-index, and the parent mapping is not updated alongside, chunks silently point to the wrong parent. Store the mapping as part of your indexing pipeline, tested like any other data integrity constraint.

Making parents too large. A "parent" that is an entire 50-page document defeats the purpose — you are back to stuffing everything into context, with a search step bolted on for nothing. Parents are usually a paragraph, a section, or a handful of sentences, not a whole document.

Deduplicating returned parents badly. If two different child chunks from the same parent both score highly, a naive implementation returns that parent's full text twice. Deduplicate on parent_id before assembling the final context.

Forgetting this doubles your storage. Every chunk now needs both its own text and a reference to the parent's text, stored somewhere retrievable. That is a real, if small, infrastructure cost worth planning for.

Try it yourself

Add a third parent paragraph about an unrelated topic, with its own children, and re-run the search for the same buyback question.

The result should not change — the new paragraph has no overlap with "buyback," so it should not outrank the correct match. If it does win, that is worth investigating: on a small demo corpus with no stop-word removal, an unrelated paragraph can occasionally out-score a real match purely by chance overlap in common words.

What to learn next

Researcher — Mathematics and papers.

Decoupling retrieval granularity from generation granularity

Parent-document retrieval is the general pattern this section has been building toward: the unit indexed for search need not equal the unit supplied to the generator. Formally, define an indexing function child: D -> C, mapping a document D to a set of small units C, and a lookup function parent: C -> P, mapping each child unit to a larger context window P. Retrieval operates entirely within C; generation consumes parent(c) for the top-scoring c. Gao et al. (2023), Retrieval-Augmented Generation for Large Language Models: A Survey (arXiv:2312.10997), groups this under post-retrieval context optimisation, distinct from the chunking decisions covered earlier in this section.

Relation to sentence-window retrieval

A closely related pattern, sometimes called sentence-window retrieval, sets child to individual sentences and parent(c) to a fixed window of k sentences on either side of c, rather than a pre-defined paragraph or section. This trades a stable, document-authored parent boundary for a uniform, tunable window size, at the cost of a returned window that can straddle two unrelated paragraphs if k is set too large.

Why small units improve retrieval precision

A bi-encoder embedding averages signal across every token in its input. A long child unit dilutes a short, specific fact with surrounding, less relevant text, lowering that fact's contribution to the resulting vector. Shrinking the indexed unit concentrates the signal, which is the same underlying mechanism explored in Karpukhin et al. (2020), Dense Passage Retrieval for Open-Domain Question Answering (arXiv:2004.04906), where passage granularity is treated as a tunable retrieval hyperparameter, not a fixed property of the corpus.

Cost analysis

Indexing cost scales with |C|, the number of child units, which is typically several times larger than the number of parent units, since each parent is split into multiple children. Storage cost includes both the child embeddings and the parent text, though the parent text can often be stored once and referenced by ID from multiple children, avoiding duplication. Query-time cost is dominated by the nearest-neighbour search over C, unaffected by parent size, since the parent lookup after retrieval is a constant-time dictionary access.

Key references

  • Karpukhin, V. et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering. arXiv:2004.04906
  • Gao, Y. et al. (2023). Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv:2312.10997

Current state and open problems

Choosing parent size — sentence-window, paragraph, section, or a fixed token budget — remains empirical, tuned against a retrieval evaluation set rather than derived from a principled model of context sufficiency. A related open question is how many distinct parents to return when top child matches span several different parents: returning too few risks omitting a relevant passage, and returning too many reintroduces the context-dilution problem this pattern exists to solve.

What to learn next