recallA retrieval
field guide

How RAG Systems Fail

Build this skill

Map RAG failures to the stage where the answer was lost, and find the first broken stage from a trace.

Great in the demo, wrong in production

A RAG system usually looks impressive in its first demo. The team asks a handful of questions they know the documents answer, the answers come back fluent and cited, and everyone is happy. Then real users arrive. They ask vague questions, use different words, ask about last year's rules, combine two topics in one sentence, or ask things the documents do not cover. Answers start going wrong in ways the demo never showed.

This gap is normal. A demo tests the system on the questions it was built for; production tests it on everything. Closing the gap starts with a clear map of how RAG systems fail, so that each failure can be traced to the stage that caused it and fixed there.

A map of failure points

Every RAG answer passes through the same chain of stages, and each stage has its own typical failures. A useful way to think about them is in three groups: the answer was never available, it was available but not retrieved, or it was retrieved but not used correctly.

Group Stage Typical failure
Never available Content The documents simply do not contain the answer
Parsing The answer was in a table or scan that was read wrongly
Freshness The index still holds an outdated version
Available, not retrieved Chunking The answer was split across chunks, or a chunk lost its context
Embedding / search The query's words or meaning did not match the right chunk
Ranking / top-K The right chunk was found but ranked below the cut-off
Retrieved, not used Context building The right chunk was dropped to fit the budget
Generation The model ignored, misread or went beyond the evidence
Format The answer was correct but incomplete, wrongly cited or badly structured
Contentis the answerthere?Parse + chunkwas it capturedwhole?Retrieve +rankwas it foundand kept?Build contextdid it reachthe prompt?Generatewas it usedfaithfully?

Each stage can lose the answer. Diagnosis means finding the first stage where it disappeared.

Research has documented the same pattern. A 2024 paper by Scott Barnett and colleagues, "Seven Failure Points When Engineering a Retrieval Augmented Generation System", drew on building several real RAG systems and listed failures closely matching this table: missing content, missed top-ranked documents, not in context, not extracted, wrong format, incorrect specificity and incomplete answers.

Finding the first broken stage

The key diagnostic habit is to find the first stage where the answer disappeared, because every later stage is working with what it was given. Four questions, asked in order, almost always identify it:

  1. Is the answer in the source documents at all? If not, it is a content gap, not a technical failure.
  2. Is it in the index, correctly parsed and chunked? Search the stored chunks directly for the key phrase.
  3. Is it in the retrieved results, and how high? Look at the top 20, not just the top 5.
  4. Is it in the final prompt, and did the answer use it faithfully? Read the actual prompt.

The program below runs these four checks against a logged trace of one failed question.

python · runs in your browser

Expected output:

text
ranking: found at rank 4, but only top 3 kept

first_broken_stage asks the same question at every stage, in order: is the key fact still here? It checks the source documents, then the indexed chunks, then the retrieved list, then whether the fact's rank was inside the top K, then the prompt. The function returns at the first stage where the answer is no, because everything after that stage never had a chance.

The model is not to blame here: the right chunk was retrieved, but only at rank 4, and the prompt kept the top 3. The fix is in ranking (a reranker, or a larger K), not in the prompt.

Practise diagnosing

Building a failure benchmark

A practical way to make failures visible is to break a working system on purpose. Create a few deliberately broken versions, such as one with tiny chunks, one with a weak embedding model, one with no grounding instruction in the prompt, and one with an outdated index, and run your test questions through each. Seeing how each kind of failure shows up in the answers, and in the metrics, trains you and your tools to recognise it in the real system.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close