Late Chunking
Explain the lost-context problem and how late chunking computes chunk embeddings with the whole document in view.
The problem with embedding chunks on their own
Take this passage from the hostel handbook, split into three chunks:
- "Kaveri Hostel Block C houses first-year students."
- "Its gates close at 10:00 PM on weekdays."
- "It also has a 24-hour reading room on the ground floor."
When each chunk is embedded separately, chunk 2 does not know what "its" refers to. Its embedding represents "some gates close at 10 PM", with no hint that this is about Block C or first-year hostels. A student who asks "when does the first-year hostel close?" may not find it, because the words that connect it to the question, "first-year" and "hostel", sit in a different chunk.
This is the lost context problem: chunking cuts the links between sentences that refer to each other through words like "it", "this policy" or "the above", and each chunk's embedding is calculated as if the rest of the document did not exist.
Embed first, chunk later
Late chunking, introduced by researchers at Jina AI in 2024, reverses the usual order. Normally you chunk first and then embed each chunk. Late chunking embeds first and chunks later:
- Run the whole document (or as much as fits) through a long-context embedding model in one go.
- The model produces a vector for every token, and because it read the whole document at once, each token's vector already reflects its context. The vector for "its" in chunk 2 carries information about Block C.
- Only now, split the token vectors at the chunk boundaries.
- Average the token vectors inside each chunk to get one embedding per chunk. This averaging is called pooling.
Late chunking lets each chunk's embedding carry context from the rest of the document.
Each chunk still gets its own embedding, and is still stored and searched exactly like an ordinary chunk. The only difference is that those embeddings were computed with the whole document in view.
Seeing the effect
The program below simulates the idea with tiny hand-made vectors. Each token has a vector, and "contextual" vectors borrow some meaning from the document's opening sentence, which is what a long-context model does when it reads everything at once.
Expected output:
chunk embedded alone: 0.74
chunk via late chunking: 0.99
The key line is late = 0.6 * alone + 0.4 * context_from_doc. It mixes the chunk's own meaning with some of the meaning of the text around it, which stands in for what a long-context model does when it reads the whole document before the chunks are cut. The weights 0.6 and 0.4 are made up; change them and watch the second score move.
The late-chunked embedding is closer to the question, because it "remembers" that the gates belong to the first-year hostel.
What it needs, and what it costs
Late chunking has a few requirements:
- A long-context embedding model that can read whole documents (often 8,000 tokens or more), such as Jina's v2 and v3 embedding models.
- Access to token-level vectors, not just one vector per input. Some embedding APIs only return the final pooled vector, which makes late chunking impossible with them.
- Documents that fit. A 200-page manual still has to be split into windows, and context only flows within each window.
In return, it costs almost nothing extra at search time: the chunks are stored and searched like any others. Indexing is a little heavier, because each document is processed as one long input.
Late chunking compared with other fixes
Lost context can be fixed in other ways too, and it helps to know how they compare.
| Approach | How it adds context | Extra cost |
|---|---|---|
| Add headings to each chunk | Prepends "Hostel Handbook › Block C" to the text | Almost none |
| Parent-child retrieval | Returns the whole parent section to the model | A second store; bigger prompts |
| Late chunking | Embeddings computed with the whole document in view | A long-context model; token vectors |
| Contextual chunk descriptions | A language model writes a short context line for each chunk | One language model call per chunk |
These can be combined. Adding headings is nearly free and should almost always be done. Late chunking improves the embeddings themselves. Parent-child retrieval improves what the model reads. As always, measure on your own test questions before adding complexity.