Recursive and Structure-aware Chunking
Split documents recursively and by headings so that chunks follow the author's structure.
Cut where the writer paused
Cutting a document every 300 tokens is simple, but it ignores everything the author did to organise the text. Writers already divide documents into sections, paragraphs and sentences, and those divisions mark where one idea ends and the next begins. A chunk that follows them is much more likely to contain one complete idea.
Structure-aware chunking uses the document's own structure to decide where to cut. Recursive chunking is the most common way to do it when the structure is only partly visible: try to split at the biggest natural break first, and only fall back to smaller breaks when a piece is still too long.
How recursive chunking works
A recursive splitter is given a list of separators, from the strongest break to the weakest. For plain text a typical list is:
- A blank line (between paragraphs)
- A new line
- A sentence ending, such as ". "
- A space (between words)
- As a last resort, between characters
It first splits the text at blank lines. Any piece that fits within the size limit becomes a chunk. Any piece that is still too long is split again at the next separator down, and so on. "Recursive" just means the same process is repeated on the pieces that are too big.
Split at the biggest break first. Only pieces that are still too long get split at a smaller break.
The result is that most chunks end at the end of a paragraph or sentence, rather than in the middle of a word or a rule.
The program below is a small recursive splitter written from scratch. Run it and look at where the cuts fall.
Expected output:
[1] (4 words) Section 4: Medical Leave
[2] (24 words) Students may take up to 10 days of medical leave in a semester. More than 10 days needs approval from the Dean of Academics.
[3] (16 words) A certificate from a registered doctor must reach the department office within 7 days of rejoining.
| Part of the code | What it does |
|---|---|
separators=("\n\n", "\n", ". ", " ") |
The cut points to try, coarsest first: paragraph breaks, line breaks, sentence ends, then spaces. |
candidate = (current + sep + piece) ... |
Tries adding the next piece to the chunk being built. If it still fits, the loop carries on. |
chunks.append(current.strip()) |
The next piece would not fit, so the chunk built so far is closed. |
recursive_split(piece, max_words, rest) |
A piece that is too big by itself is split again with the next, finer separator. This is the recursive step. |
Each chunk is a whole sentence or paragraph. None is cut mid-sentence. Notice, though, that the heading became a tiny chunk of its own, separated from the rules it introduces. That is a common weakness of plain recursive splitting, and the next idea fixes it.
Using real structure: headings, Markdown and code
When documents have been parsed into Markdown, the structure is visible: lines starting with # are headings. A heading-aware chunker splits at headings first, so each chunk covers one section, and it copies the heading path into every chunk, such as "Academic Regulations › Section 4: Medical Leave". Even a small chunk from deep inside the section then says what it is about.
Different content types have their own natural boundaries:
| Content | Natural split points |
|---|---|
| Markdown or HTML documents | Headings, then paragraphs |
| FAQs | One question and its answer per chunk |
| Legal documents and policies | Numbered clauses and sub-clauses |
| Code | Functions and classes |
| Tables | Keep each table whole, or one row per chunk with its headers |
Compare three splitting methods on the same text in the lab below. Notice where each one cuts, and whether any chunk holds the whole answer.
Doing it with a library
Frameworks provide ready-made splitters, so you rarely write your own. LangChain's text splitters package includes a recursive splitter and a Markdown header splitter. This example runs on your own computer after installing the package:
# pip install langchain-text-splitters
from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter
# 1. Split at headings, keeping the heading names as metadata
by_heading = MarkdownHeaderTextSplitter(headers_to_split_on=[("#", "section"), ("##", "subsection")])
sections = by_heading.split_text(markdown_text)
# 2. Split any long section further, at paragraphs, then sentences, then words
splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=100)
chunks = splitter.split_documents(sections)
The two-step pattern, headings first and then recursive splitting within long sections, is a strong default for most document collections.
When structure-aware chunking helps most
Structure-aware chunking helps most on long, well-organised documents such as regulations, manuals and technical documentation, where headings carry meaning. It helps least on text with no structure, such as a transcript of a meeting or a scanned letter with no headings, where you are back to sentences and sizes. Either way, always test on your own questions: the method that looks neatest is not always the one that retrieves best.