Learn AI Create the Future — with Pranay
Main site

Chunking and Long Documents

Keeping the trail back to the source

Provenance is the record every chunk carries back to its exact source, so an answer built from it can be checked, cited, and trusted.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. Where you have already seen it
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Provenance is the record a chunk carries back to its exact source — which document, which page, which words.

Think about a family photo album where every photo has a sticky note on the back. Which trip, which year, which roll of film it was developed from. Pull one photo out of the album, and the note still tells you where it belongs.

Why it exists

A chunk answering a question is only as trustworthy as its source. If a chatbot says "refunds take 7 business days," someone needs to check where that came from. Which policy document? Updated when, on what page?

Without provenance, a chunk becomes floating text the moment it leaves the document. When two chunks disagree, there is no way to tell which is newer, or more authoritative, without digging through the original files by hand.

Storing provenance alongside every chunk, from the moment it is created, means that checking work later costs nothing extra. Reconstructing it after the fact is far harder, and sometimes impossible if the original files have since changed.

How it works

Document: "annual-report-2024.pdf"

Chunk stored with its provenance:

  text:        "Revenue grew 18 percent to 210 crore rupees."
  doc_id:      "annual-report-2024"
  page:        1
  char_start:  20
  char_end:    64

Citation shown to the user:
  "annual-report-2024, page 1, characters 20-64"

Anyone can now jump straight to the exact spot in the original document the chunk came from.

Where you have already seen it

  • Wikipedia's citation footnotes. Every fact links back to a specific source, so a claim can be checked.
  • Google Books' "page X" links. A search result opens a scanned book at the exact page a phrase appeared on.
  • AI chat tools that cite sources, such as Perplexity, showing which web page each part of an answer came from.

Remember this

  • Provenance means storing exactly where a chunk came from: document, page, and position.
  • It has to be captured when the chunk is created, or it becomes very hard to recover.
  • Good provenance turns "trust the AI" into "check the AI," which is a far safer thing to build on.

What to learn next

Developer — Code and libraries.

This attaches document ID, page, and character offsets to each chunk, then verifies the offsets actually point back to the right text.

Setup

bash
python --version   # 3.9 or newer, only the standard library is used

Chunks that carry their own citation

provenance.py
from dataclasses import dataclass

@dataclass
class Chunk:
    text: str
    doc_id: str
    page: int
    char_start: int
    char_end: int

    def citation(self):
        return f"{self.doc_id}, page {self.page}, chars {self.char_start}-{self.char_end}"

document_text = (
    "Annual Report 2024. Revenue grew 18 percent to 210 crore rupees. "
    "Operating costs rose 9 percent, mainly from hiring in engineering."
)

def make_chunks(text, doc_id, page, size=60, overlap=10):
    chunks = []
    start = 0
    while start < len(text):
        end = min(start + size, len(text))
        chunks.append(Chunk(text[start:end], doc_id, page, start, end))
        if end == len(text):
            break
        start = end - overlap
    return chunks

chunks = make_chunks(document_text, doc_id="annual-report-2024", page=1)

for c in chunks:
    print(c.citation())
    print(" ", repr(c.text))

# Verify the promise: does the offset really point back to this text?
c = chunks[1]
recovered = document_text[c.char_start:c.char_end]
print()
print("recovered text matches stored chunk text:", recovered == c.text)
Output
annual-report-2024, page 1, chars 0-60
  'Annual Report 2024. Revenue grew 18 percent to 210 crore rup'
annual-report-2024, page 1, chars 50-110
  ' crore rupees. Operating costs rose 9 percent, mainly from h'
annual-report-2024, page 1, chars 100-131
  'nly from hiring in engineering.'

recovered text matches stored chunk text: True

Line by line

@dataclass gives Chunk a constructor and a clean repr for free, without writing __init__ by hand. Every field needed for a full citation — doc_id, page, char_start, char_end — travels with the chunk automatically.

citation() turns the stored fields into a human-readable string. This is the exact text a "source: ..." footnote in a chat interface would show.

The verification step matters more than it looks. document_text[c.char_start:c.char_end] == c.text proves the stored offsets are not decoration — they genuinely point back to the source text. A provenance system that fails this check is lying about where its answers come from.

Common mistakes

Storing offsets against a version of the document that later changes. If the source PDF gets re-uploaded with edits, old offsets can point at the wrong text, or past the end of the file entirely. Store a content hash or version ID alongside the offsets, and re-chunk when the source changes.

Computing offsets after cleaning the text, then storing them against the raw file. If whitespace normalisation or HTML-tag removal happens before chunking, offsets must be tracked against that cleaned version consistently — never mixed with raw-file offsets.

Treating page as reliable for a re-flowable format. A PDF has fixed pages. An HTML page, or a Word document viewed at a different zoom level, does not. Page numbers are meaningful provenance for fixed-layout formats and misleading for reflowable ones — use character or paragraph offsets there instead.

Losing provenance during chunk overlap. When overlap is nonzero, as here, some characters appear in two chunks. Deduplicating overlapping citations before showing them to a user avoids listing the "same" source twice under two different offset ranges.

Try it yourself

Change overlap to 0 and re-run. Look at where chunks[0] ends and chunks[1] begins.

With zero overlap, the character ranges become adjacent with no shared text — chunk 0 ends where chunk 1 starts, exactly. That is easy to verify with real numbers, and it is worth checking directly against the size=60 value used above.

What to learn next

Researcher — Mathematics and papers.

Provenance as a first-class retrieval artefact

Formally, a provenance-aware chunking function is not chunk: D -> Text*, mapping a document to plain text pieces, but chunk: D -> (Text, Metadata)*, where Metadata includes at minimum a document identifier, a byte or character offset range, and — for versioned corpora — a content hash or revision ID. Treating provenance as part of the chunk's type, not an optional side table, makes it far harder to accidentally drop during re-indexing or migration.

Attribution versus citation

The RAG literature distinguishes two related but separate properties. Provenance (this lesson) is the mechanical record of where a chunk's text came from, checkable independent of any model output. Attribution — covered in Checking that an answer really came from the source — is the separate claim that a specific generated answer is actually supported by a specific cited chunk. Provenance without attribution checking is common, and insufficient on its own: a system can cite a real, correctly-tracked source that does not actually support the generated claim. WebGPT (Nakano et al., 2021, arXiv:2112.09332) was an early large-scale system built explicitly around this distinction, training a model to browse and quote sources with real, checkable citations rather than free-form, unsourced claims.

Storage design

Two common schemas exist. Inline metadata, storing offsets directly on the chunk record, as in the developer block's Chunk dataclass, is simple and colocated. Separate provenance store, keeping a lookup table from chunk ID to source location, trades a join at read time for the ability to update provenance — for instance, after a document is corrected — without touching the embedding index. Large production RAG systems typically use the second pattern, since re-embedding on every metadata correction is expensive and unnecessary.

Cost

Storing offsets is O(1) additional space per chunk — a few integers and a string ID. Verifying an offset against source text, as in the developer block's check, is O(chunk length). Neither is a meaningful cost against the embedding and index storage a retrieval system already pays for. There is close to no excuse, on cost grounds, for a production RAG system to omit provenance entirely.

Key references

  • Nakano, R. et al. (2021). WebGPT: Browser-assisted question-answering with human feedback. arXiv:2112.09332
  • Gao, Y. et al. (2023). Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv:2312.10997

Current state and open problems

Provenance tracking is largely a solved engineering problem — the difficulty is organisational discipline, not algorithmic novelty. What remains open is provenance across derived content: a chunk generated by summarising several source passages, rather than extracted directly, has no single clean offset range to point to. Multi-source attribution, where a claim genuinely synthesises several documents, is an active and unresolved area, closely tied to the multi-hop question answering problem covered later in this batch of lessons.

What to learn next