recallA retrieval
field guide

Debugging Chunking and Embeddings

Build this skill

Diagnose fragmentation, missing context, language mismatch, missing prefixes and truncation in chunks and embeddings.

Many retrieval failures are decided long before a question is asked, in two indexing choices: how documents were chunked and which embedding model turned chunks into vectors. When these go wrong, every later stage works correctly on bad input, and the symptoms (missing answers, half answers, irrelevant results) look like search or model problems. Debugging them means looking at the chunks and embeddings directly.

Chunking problemsAnswer split across chunksChunk lost its headingSeveral topics in one chunkTables without headersEmbedding problemsModel weak in the languageMissing query/passage prefixesLong chunks cut offOld and new vectors mixed

Both kinds of problem are fixed at indexing time, and both look like search failures from the outside.

Chunking problems and their symptoms

Problem What it looks like in a chunk Symptom in answers
Fragmentation "more than 10 days needs" / "approval from the Dean" in separate chunks Half answers; "sources don't say who approves"
Missing context "The limit is 10 days." with no subject The right chunk ranks low for the obvious question
Mixed topics Leave rules and hostel rules in one large chunk Blurry matches; irrelevant text in the prompt
Broken tables Rows of numbers without headers Wrong numbers in answers
Boilerplate Every chunk begins "Kaveri Institute of Technology · Confidential" Unrelated chunks look similar

The fastest diagnostic is simply to read chunks: pick a document that produced a bad answer and print the chunks it was split into. Problems like fragmentation and missing headings are obvious to a human eye in seconds.

A quantitative check is to take your test questions, find the chunks that contain each reference answer, and count how often the answer is split across more than one chunk. The program below does this for two chunking settings.

python · runs in your browser

Expected output:

text
fixed 12 words       4 chunks, answers split across chunks: 2 of 2
one sentence each    4 chunks, answers split across chunks: 1 of 2
two sentences each   2 chunks, answers split across chunks: 1 of 2

fixed_chunks cuts every 12 words and sentence_chunks cuts at full stops. For each chunking, the line that computes split goes through the two test answers and counts those for which no single chunk contains both of the facts the answer needs. A lower count means fewer answers that depend on retrieving two chunks at once.

Embedding problems and their symptoms

Embedding problems show up as right chunk exists, but similarity search does not find it. Common causes:

  • Domain or language mismatch. A general English model may not place "backlog", "re-evaluation" or Telugu queries well.
  • Missing query or passage prefixes. Some models expect "query: " and "passage: "; leaving them out quietly lowers quality.
  • Truncation. Chunks longer than the model's input limit are cut off, so their endings are never embedded.
  • Mixed models. Some chunks were embedded with an old model and some with a new one, so their vectors are not comparable.
  • Wrong distance measure. The index uses a different similarity measure from the one the model was trained for.

A powerful check is to swap the embedding model and re-run the same test set, keeping everything else fixed. If Recall@K jumps, the embedding model was the bottleneck. Look at which questions changed: often a model is much better for one language or one kind of question.

Looking at the embedding space

Sometimes it helps to look at the embeddings themselves. Two simple checks:

  • Nearest neighbours of a chunk. Pick a chunk and list the chunks closest to it. If the neighbours of a medical leave rule are library rules, something is wrong (often boilerplate, or truncation of the actual content).
  • Similarity between near-duplicates. Two versions of the same rule should be very similar; if not, parsing may have damaged one of them.

Dimensionality-reduction plots (such as UMAP or t-SNE, which squash embeddings down to two dimensions for drawing) can show whether chunks cluster by topic. They are good for spotting broad problems, such as all chunks from one badly parsed document forming a strange separate cluster, but they distort distances, so do not read fine detail into them.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close