recallA retrieval
field guide

Semantic Chunking

Build this skill

Find topic boundaries with sentence embeddings and judge when semantic chunking is worth its cost.

Cutting where the topic changes

Recursive chunking cuts at paragraphs and sentences, which is a good guess about where ideas end. But not all documents are well organised. A long circular may run several topics together in one paragraph; a meeting transcript has no paragraphs at all; a scanned notice may have lost its formatting. In these cases, punctuation tells you little about where one idea ends.

Semantic chunking tries to cut where the meaning changes. It uses embeddings to measure how similar each sentence is to the next, and starts a new chunk when the topic shifts.

How it works

The method has four steps:

  1. Split the text into sentences.
  2. Embed each sentence.
  3. Measure the similarity between each sentence and the next one.
  4. Where the similarity drops sharply, cut. That drop is called a breakpoint.

Consecutive sentences about the same topic, say three sentences about medical leave, have similar embeddings. When the text moves on to hostel timings, the next sentence's embedding points in a different direction, and the similarity falls. The chunker places a boundary there.

S1 → S2 (leave → leave)0.86S2 → S3 (leave → leave)0.81S3 → S4 (leave → hostel)0.22 ← cut hereS4 → S5 (hostel → hostel)0.79S5 → S6 (hostel → hostel)0.83Similarity between each pair of neighbouring sentences (illustrative)

Similarity stays high within a topic and drops where the topic changes. The drop marks a breakpoint.

How big a drop counts as a breakpoint is a setting. Common choices are a fixed threshold (cut whenever similarity falls below, say, 0.5), or a relative one (cut at the biggest drops in the document, for example the lowest 5 percent of similarities). A lower threshold produces fewer, longer chunks; a higher one produces more, shorter chunks.

A small semantic chunker

The program below implements the idea. Because real embedding models cannot run inside this page, each sentence has a small hand-made embedding; the chunking logic is exactly what you would use with real embeddings.

python · runs in your browser

Expected output:

text
[1] Students may take up to 10 days of medical leave. Extra days need approval from the Dean of Academics. A doctor's certificate is required within 7 days.
[2] Hostel gates close at 10 PM for first-years. Late entry needs written permission from the warden.
[3] Library books can be borrowed for 14 days.
Line What it does
zip(sentences, sentences[1:]) Pairs every sentence with the one after it.
sim = cos(prev_vec, vec) How similar the two neighbouring sentences are.
if sim < threshold A sharp drop means the topic changed: the current chunk is closed and a new one starts.
current.append(text) Otherwise the sentence joins the chunk being built.

The chunker found three topics, leave, hostel and library, without being told where the paragraphs were. Move the threshold yourself to see how sensitive the result is.

Fixed, recursive or semantic?

Semantic chunking is appealing, but it is not automatically better. Compare the trade-offs:

Fixed size Recursive / structure-aware Semantic
Cuts at Every N tokens Headings, paragraphs, sentences Changes in meaning
Needs embeddings to chunk No No Yes, for every sentence
Speed and cost Fastest Fast Slowest
Chunk sizes Even Fairly even Can vary a lot
Best for Quick baselines Well-structured documents Unstructured text, mixed topics

Two cautions come up often in practice. First, semantic chunking costs more, because every sentence must be embedded just to decide the cuts, before the chunks are embedded again for search. Second, it can produce very uneven chunks, from a single short sentence to a huge block, so most implementations also enforce a minimum and maximum size. Several published comparisons have found that on well-structured documents, a good recursive splitter performs about as well as semantic chunking at a fraction of the cost.

The sensible approach is the same as for chunk size: try it on your own documents and test questions, and only keep it if it measurably helps.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close