recallA retrieval
field guide

Chunk Size and Overlap

Build this skill

Explain the chunk size trade-off, use overlap sensibly and choose a size by measuring on test questions.

Why documents are split into chunks

A retrieval system rarely searches whole documents. It splits each document into smaller passages called chunks, embeds each chunk separately, and searches over the chunks. There are three good reasons for this.

First, precision. A 42-page regulations PDF covers attendance, leave, exams, fees and discipline. One embedding for the whole document would be a vague blend of all of them, close to nothing in particular. A chunk about medical leave has a sharp embedding that sits right next to questions about medical leave.

Second, model limits. Embedding models stop reading after a fixed number of tokens, so long documents must be split anyway.

Third, prompt space. A language model can only read so much at once, and every extra token costs money and time. Sending the three most relevant paragraphs is far better than sending three whole documents.

How you split, though, makes a big difference. Chunking is one of the decisions that most affects RAG quality, and the right answer depends on the documents.

The size trade-off

Small chunks (a sentence or two) give very precise matches, but each chunk may be missing context. A chunk that says "More than 10 days needs approval from the Dean of Academics" does not say that it is about medical leave.

Large chunks (several paragraphs) keep context together, but their embeddings are blurrier, they match more loosely, and they use up prompt space with text the question did not need.

Small chunksPrecise matchesSharp embeddingsLess prompt space usedMay lose contextAnswer may be split apartLarge chunksContext kept togetherWhole answer in one pieceBlurrier embeddingsMore prompt space usedMore unrelated text

There is no universal best size. The right size depends on the documents and the questions.

Chunk size is usually measured in tokens (or sometimes characters or words). Common starting points are somewhere between 200 and 800 tokens, but this is only a starting point. Short FAQ entries may work best as one chunk each; long technical manuals may need larger chunks to keep procedures together.

Overlap: a safety margin at the edges

When you cut text every N words, some cuts land in awkward places, such as the middle of the sentence that holds the answer. Overlap means each chunk repeats the last few words of the previous chunk, so that text near a boundary appears in two chunks and is less likely to be cut in half. A common choice is an overlap of 10 to 20 percent of the chunk size.

Overlap has a cost: more chunks to store and embed, and near-duplicate results when two overlapping chunks both match. It is a safety margin, not a cure.

Try it below. The question needs two facts: the limit ("10 days") and who approves extra days ("Dean of Academics"). Change the chunk size and overlap, and watch whether any single chunk contains both.

Measuring instead of guessing

Because the best size depends on your documents and questions, the reliable way to choose is to measure. Take a set of test questions with known answer passages, chunk the collection at a few sizes, and check how often the right chunk appears in the top results. The program below does a tiny version: for each size, it checks whether a single chunk contains both facts the question needs.

python · runs in your browser

Expected output:

text
size 10, overlap 0: 6 chunks, answer in one chunk: False
size 20, overlap 0: 3 chunks, answer in one chunk: True
size 20, overlap 5: 4 chunks, answer in one chunk: False
size 30, overlap 0: 2 chunks, answer in one chunk: False
size 50, overlap 0: 2 chunks, answer in one chunk: True

In chunk, step = size - overlap is how far the window moves each time. With size 20 and overlap 5, each chunk starts 15 words after the previous one, so its first 5 words repeat the last 5 of the chunk before. The line with any(all(...)) asks whether any single chunk contains both of the needed phrases.

The results are not a smooth curve. Size 20 happened to keep both facts together, size 30 split them again, and adding a 5-word overlap at size 20 moved the cut points so that the facts were split. Where a fixed-size cut lands is partly luck, which is exactly why it pays to measure on real questions, and to cut at natural boundaries such as sentences and headings. With real documents and real questions, the same experiment tells you which size gives the best Recall@K, and whether overlap helps enough to be worth the extra chunks.

Practical guidance

  • Start with the document's own structure where you can, splitting at headings and paragraphs rather than at fixed word counts.
  • Pick a size that fits your questions. Short factual questions suit smaller chunks; "explain the whole procedure" questions suit larger ones.
  • Keep the heading with the chunk. Adding "Section 4: Medical Leave" to the start of every chunk from that section recovers much of the context small chunks lose.
  • Test two or three sizes on real questions before committing, and re-test when the kind of documents changes.
  • Record the chunking settings with every chunk, because changing them later means re-chunking and re-embedding everything.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close