recallA retrieval
field guide

Late Chunking

Build this skill

Explain the lost-context problem and how late chunking computes chunk embeddings with the whole document in view.

The problem with embedding chunks on their own

Take this passage from the hostel handbook, split into three chunks:

  1. "Kaveri Hostel Block C houses first-year students."
  2. "Its gates close at 10:00 PM on weekdays."
  3. "It also has a 24-hour reading room on the ground floor."

When each chunk is embedded separately, chunk 2 does not know what "its" refers to. Its embedding represents "some gates close at 10 PM", with no hint that this is about Block C or first-year hostels. A student who asks "when does the first-year hostel close?" may not find it, because the words that connect it to the question, "first-year" and "hostel", sit in a different chunk.

This is the lost context problem: chunking cuts the links between sentences that refer to each other through words like "it", "this policy" or "the above", and each chunk's embedding is calculated as if the rest of the document did not exist.

Embed first, chunk later

Late chunking, introduced by researchers at Jina AI in 2024, reverses the usual order. Normally you chunk first and then embed each chunk. Late chunking embeds first and chunks later:

  1. Run the whole document (or as much as fits) through a long-context embedding model in one go.
  2. The model produces a vector for every token, and because it read the whole document at once, each token's vector already reflects its context. The vector for "its" in chunk 2 carries information about Block C.
  3. Only now, split the token vectors at the chunk boundaries.
  4. Average the token vectors inside each chunk to get one embedding per chunk. This averaging is called pooling.
Usual: chunk, then embedSplit text into chunksEmbed chunk 1 on its ownEmbed chunk 2 on its own“its gates” loses Block CContext across chunks is lostLate: embed, then chunkEmbed the whole documentEvery token sees every otherSplit token vectors at chunk edgesAverage within each chunk“its gates” still knows Block C

Late chunking lets each chunk's embedding carry context from the rest of the document.

Each chunk still gets its own embedding, and is still stored and searched exactly like an ordinary chunk. The only difference is that those embeddings were computed with the whole document in view.

Seeing the effect

The program below simulates the idea with tiny hand-made vectors. Each token has a vector, and "contextual" vectors borrow some meaning from the document's opening sentence, which is what a long-context model does when it reads everything at once.

python · runs in your browser

Expected output:

text
chunk embedded alone:   0.74
chunk via late chunking: 0.99

The key line is late = 0.6 * alone + 0.4 * context_from_doc. It mixes the chunk's own meaning with some of the meaning of the text around it, which stands in for what a long-context model does when it reads the whole document before the chunks are cut. The weights 0.6 and 0.4 are made up; change them and watch the second score move.

The late-chunked embedding is closer to the question, because it "remembers" that the gates belong to the first-year hostel.

What it needs, and what it costs

Late chunking has a few requirements:

  • A long-context embedding model that can read whole documents (often 8,000 tokens or more), such as Jina's v2 and v3 embedding models.
  • Access to token-level vectors, not just one vector per input. Some embedding APIs only return the final pooled vector, which makes late chunking impossible with them.
  • Documents that fit. A 200-page manual still has to be split into windows, and context only flows within each window.

In return, it costs almost nothing extra at search time: the chunks are stored and searched like any others. Indexing is a little heavier, because each document is processed as one long input.

Late chunking compared with other fixes

Lost context can be fixed in other ways too, and it helps to know how they compare.

Approach How it adds context Extra cost
Add headings to each chunk Prepends "Hostel Handbook › Block C" to the text Almost none
Parent-child retrieval Returns the whole parent section to the model A second store; bigger prompts
Late chunking Embeddings computed with the whole document in view A long-context model; token vectors
Contextual chunk descriptions A language model writes a short context line for each chunk One language model call per chunk

These can be combined. Adding headings is nearly free and should almost always be done. Late chunking improves the embeddings themselves. Parent-child retrieval improves what the model reads. As always, measure on your own test questions before adding complexity.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close