Debugging Retrieval
Instrument the retriever and use a step-by-step checklist to tell recall, filter and ranking failures apart.
Don't blame the model first
When a RAG answer is wrong, the language model is the most visible part, so it gets blamed first. Teams rewrite the prompt, switch to a bigger model, and the answer is still wrong, because the model never received the right evidence. In practice, a large share of wrong answers come from retrieval. Debugging retrieval means looking at what was retrieved, with its scores and filters, before touching the prompt.
Two kinds of retrieval failure
Low recall: the right chunk is missing from the results. The model cannot use evidence it never sees, so the answer is incomplete, says "I could not find this", or is invented.
Low precision: the right chunk is there, but surrounded by irrelevant or misleading chunks, or ranked below them. The model may pick up the wrong fact (the staff leave rule instead of the student one) or produce a muddled answer.
Each has typical causes:
| Symptom | Common causes |
|---|---|
| Right chunk not in the top 20 | Vocabulary gap; weak embeddings; chunking split the answer; a filter excluded it |
| Right chunk at rank 8, K = 5 | Ranking is rough; no reranker; near-duplicates fill the top |
| Wrong-audience chunks rank high | Missing access or audience filter |
| Outdated chunk ranks above the current one | No status/version filter |
| Fewer than K results | Post-filtering removed most candidates |
Missing evidence and noisy evidence are different problems with different fixes.
Instrument the retriever
To debug retrieval you need to see inside it. A well-instrumented retriever logs, for every query:
- the query as searched (after any rewriting);
- the filters applied;
- every candidate with its ID, source, score from each method (BM25, vector) and rank before and after reranking;
- which candidates made it into the prompt.
With this, most failures can be diagnosed in a minute. The program below prints such a log for one question and highlights where the right chunk was.
Expected output:
query: How many days of medical leave can a student take?
filters: {'status': 'current', 'access': 'students'}
chunk source vec bm25# final# note
att2.1 Attendance rules 0.84 3 3 IN PROMPT
hr7.2 Staff leave policy 0.83 2 - filtered out
regs4.1 Regulations 2026 §4.1 0.81 1 1 IN PROMPT <- relevant
regs25 Regulations 2025 §4.1 0.81 4 - filtered out
lib2 Library rules 0.62 9 4
Each row is one candidate chunk, with what every stage did to it: its vector score, its keyword rank, its rank after reranking, and whether it passed the filters. The note column is built from two tests, so it reads "filtered out" for a chunk that failed a filter and "IN PROMPT" for one that finished inside the top K.
The log shows the system behaving correctly: filters removed the staff policy and the outdated rule, and reranking moved the right chunk to first place. If the answer was still wrong, the problem is after retrieval.
A step-by-step checklist
When a question fails, work through these checks in order:
- Is the right chunk in the index? Search the stored chunks for a key phrase. If not, it is an ingestion problem.
- Did a filter exclude it? Run the search without filters. If the chunk appears, check its metadata (wrong access level, wrong status, misspelt department).
- Where does it rank in the top 20 or 50? If absent, try BM25 and vector search separately: if one finds it, add hybrid search; if neither does, look at vocabulary, embeddings and chunking.
- Is it ranked just below K? Consider a reranker, a larger K, or MMR if near-duplicates are crowding it out.
- Is it in the prompt? If yes, retrieval worked, and the problem is generation.
Fix at the right level
Once the cause is found, fix it where it starts. A missing chunk is an ingestion fix. A metadata error is a data fix. A vocabulary gap is a hybrid-search or query-rewriting fix. A ranking problem is a reranking fix. And after every fix, re-run the full test set, not just the failing question, to make sure the change did not break something else.