recallA retrieval
field guide

Debugging Retrieval

Build this skill

Instrument the retriever and use a step-by-step checklist to tell recall, filter and ranking failures apart.

Don't blame the model first

When a RAG answer is wrong, the language model is the most visible part, so it gets blamed first. Teams rewrite the prompt, switch to a bigger model, and the answer is still wrong, because the model never received the right evidence. In practice, a large share of wrong answers come from retrieval. Debugging retrieval means looking at what was retrieved, with its scores and filters, before touching the prompt.

Two kinds of retrieval failure

Low recall: the right chunk is missing from the results. The model cannot use evidence it never sees, so the answer is incomplete, says "I could not find this", or is invented.

Low precision: the right chunk is there, but surrounded by irrelevant or misleading chunks, or ranked below them. The model may pick up the wrong fact (the staff leave rule instead of the student one) or produce a muddled answer.

Each has typical causes:

Symptom Common causes
Right chunk not in the top 20 Vocabulary gap; weak embeddings; chunking split the answer; a filter excluded it
Right chunk at rank 8, K = 5 Ranking is rough; no reranker; near-duplicates fill the top
Wrong-audience chunks rank high Missing access or audience filter
Outdated chunk ranks above the current one No status/version filter
Fewer than K results Post-filtering removed most candidates
Low recallRight chunk missing from resultsAnswer incomplete or “not found”Fix: hybrid, rewriting, chunkingLow precisionRight chunk buried among wrong onesWrong fact picked, muddled answerFix: filters, reranking, MMR

Missing evidence and noisy evidence are different problems with different fixes.

Instrument the retriever

To debug retrieval you need to see inside it. A well-instrumented retriever logs, for every query:

  • the query as searched (after any rewriting);
  • the filters applied;
  • every candidate with its ID, source, score from each method (BM25, vector) and rank before and after reranking;
  • which candidates made it into the prompt.

With this, most failures can be diagnosed in a minute. The program below prints such a log for one question and highlights where the right chunk was.

python · runs in your browser

Expected output:

text
query:   How many days of medical leave can a student take?
filters: {'status': 'current', 'access': 'students'}

chunk    source                     vec  bm25#  final#  note
att2.1   Attendance rules          0.84      3       3  IN PROMPT
hr7.2    Staff leave policy        0.83      2       -  filtered out
regs4.1  Regulations 2026 §4.1     0.81      1       1  IN PROMPT  <- relevant
regs25   Regulations 2025 §4.1     0.81      4       -  filtered out
lib2     Library rules             0.62      9       4  

Each row is one candidate chunk, with what every stage did to it: its vector score, its keyword rank, its rank after reranking, and whether it passed the filters. The note column is built from two tests, so it reads "filtered out" for a chunk that failed a filter and "IN PROMPT" for one that finished inside the top K.

The log shows the system behaving correctly: filters removed the staff policy and the outdated rule, and reranking moved the right chunk to first place. If the answer was still wrong, the problem is after retrieval.

A step-by-step checklist

When a question fails, work through these checks in order:

  1. Is the right chunk in the index? Search the stored chunks for a key phrase. If not, it is an ingestion problem.
  2. Did a filter exclude it? Run the search without filters. If the chunk appears, check its metadata (wrong access level, wrong status, misspelt department).
  3. Where does it rank in the top 20 or 50? If absent, try BM25 and vector search separately: if one finds it, add hybrid search; if neither does, look at vocabulary, embeddings and chunking.
  4. Is it ranked just below K? Consider a reranker, a larger K, or MMR if near-duplicates are crowding it out.
  5. Is it in the prompt? If yes, retrieval worked, and the problem is generation.

Fix at the right level

Once the cause is found, fix it where it starts. A missing chunk is an ingestion fix. A metadata error is a data fix. A vocabulary gap is a hybrid-search or query-rewriting fix. A ranking problem is a reranking fix. And after every fix, re-run the full test set, not just the failing question, to make sure the change did not break something else.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close