Semantic Chunking
Find topic boundaries with sentence embeddings and judge when semantic chunking is worth its cost.
Cutting where the topic changes
Recursive chunking cuts at paragraphs and sentences, which is a good guess about where ideas end. But not all documents are well organised. A long circular may run several topics together in one paragraph; a meeting transcript has no paragraphs at all; a scanned notice may have lost its formatting. In these cases, punctuation tells you little about where one idea ends.
Semantic chunking tries to cut where the meaning changes. It uses embeddings to measure how similar each sentence is to the next, and starts a new chunk when the topic shifts.
How it works
The method has four steps:
- Split the text into sentences.
- Embed each sentence.
- Measure the similarity between each sentence and the next one.
- Where the similarity drops sharply, cut. That drop is called a breakpoint.
Consecutive sentences about the same topic, say three sentences about medical leave, have similar embeddings. When the text moves on to hostel timings, the next sentence's embedding points in a different direction, and the similarity falls. The chunker places a boundary there.
Similarity stays high within a topic and drops where the topic changes. The drop marks a breakpoint.
How big a drop counts as a breakpoint is a setting. Common choices are a fixed threshold (cut whenever similarity falls below, say, 0.5), or a relative one (cut at the biggest drops in the document, for example the lowest 5 percent of similarities). A lower threshold produces fewer, longer chunks; a higher one produces more, shorter chunks.
A small semantic chunker
The program below implements the idea. Because real embedding models cannot run inside this page, each sentence has a small hand-made embedding; the chunking logic is exactly what you would use with real embeddings.
Expected output:
[1] Students may take up to 10 days of medical leave. Extra days need approval from the Dean of Academics. A doctor's certificate is required within 7 days.
[2] Hostel gates close at 10 PM for first-years. Late entry needs written permission from the warden.
[3] Library books can be borrowed for 14 days.
| Line | What it does |
|---|---|
zip(sentences, sentences[1:]) |
Pairs every sentence with the one after it. |
sim = cos(prev_vec, vec) |
How similar the two neighbouring sentences are. |
if sim < threshold |
A sharp drop means the topic changed: the current chunk is closed and a new one starts. |
current.append(text) |
Otherwise the sentence joins the chunk being built. |
The chunker found three topics, leave, hostel and library, without being told where the paragraphs were. Move the threshold yourself to see how sensitive the result is.
Fixed, recursive or semantic?
Semantic chunking is appealing, but it is not automatically better. Compare the trade-offs:
| Fixed size | Recursive / structure-aware | Semantic | |
|---|---|---|---|
| Cuts at | Every N tokens | Headings, paragraphs, sentences | Changes in meaning |
| Needs embeddings to chunk | No | No | Yes, for every sentence |
| Speed and cost | Fastest | Fast | Slowest |
| Chunk sizes | Even | Fairly even | Can vary a lot |
| Best for | Quick baselines | Well-structured documents | Unstructured text, mixed topics |
Two cautions come up often in practice. First, semantic chunking costs more, because every sentence must be embedded just to decide the cuts, before the chunks are embedded again for search. Second, it can produce very uneven chunks, from a single short sentence to a huge block, so most implementations also enforce a minimum and maximum size. Several published comparisons have found that on well-structured documents, a good recursive splitter performs about as well as semantic chunking at a fraction of the cost.
The sensible approach is the same as for chunk size: try it on your own documents and test questions, and only keep it if it measurably helps.