recallA retrieval
field guide

Where Naïve RAG Breaks

Build this skill

Recognise the five common failure patterns of naïve RAG and map each to a family of fixes.

A pipeline that works, until it doesn't

A naïve RAG system, built with care, answers most simple questions well: "what is the library fine?", "when do hostel gates close?". It is when questions become harder, or documents messier, that it starts failing. The failures are rarely random. They fall into a handful of patterns, and each pattern points to a different improvement. Recognising the pattern is the first step to choosing the right fix, rather than trying every advanced technique at once.

Five common failure patterns

Missing context in a chunk

The chunk that holds the fact does not say what it is about. "The limit is 10 days" was cut away from the heading "Medical leave", so a search for medical leave ranks it low.

The answer is spread across places

"Who approves leave beyond the limit, and what paperwork is needed?" needs the approval rule and the certificate rule, which sit in different chunks, and only one of them makes the top K.

Multi-hop questions

"Is the warden of the hostel where first-years stay also on the anti-ragging committee?" needs one fact (which hostel first-years stay in), then another (who its warden is), then a third (who is on the committee). Each fact is retrievable on its own, but no single search connects them.

Aggregation questions

"How many circulars about exam dates were issued this year?" or "summarise all the changes to hostel rules" need information from many documents at once. Retrieving the top 5 chunks cannot count or summarise a whole collection.

Conflicting or outdated sources

Last year's and this year's rules are both retrieved, and nothing tells the model which one applies.

Missing contextchunk lost its headingfix: contextual chunksSpread outfacts in many chunksfix: better chunking, multi-queryMulti-hopfact → fact → factfix: agents, knowledge graphsAggregationcount or summarise allfix: GraphRAG, SQL, long contextConflictsold vs new versionsfix: metadata, freshness

Each failure pattern has its own family of fixes. The first two are about chunking and retrieval; the next two need different architectures.

Seeing the failures

Look at how one naïve system handles five hard questions below. For each one, decide which pattern it shows before reading the explanation.

Diagnose before you fix

The two families of failure need different responses.

The retrieval-quality failures (missing context, spread-out answers, conflicts) usually yield to improvements inside the existing pipeline: better chunking with headings or context added, hybrid search, reranking, multi-query retrieval, and good metadata for versions.

The structural failures (multi-hop and aggregation) are harder. A pipeline that retrieves a few similar chunks for each question is simply the wrong shape for them. They call for different architectures: agents that search in several steps, knowledge graphs that store relationships explicitly, structured queries over tables, or reading much more text at once.

The program below runs a tiny failure benchmark: for each test question it records which pattern it belongs to and whether the naïve system answered correctly, then counts failures by pattern. A table like this, built from your own questions, tells you where to invest first.

python · runs in your browser

Expected output:

text
simple lookup          1/4 failed
missing context        1/2 failed
spread out             2/3 failed
multi-hop              2/2 failed
aggregation            1/1 failed
conflicting versions   1/2 failed

overall accuracy: 6/14

Counter tallies how many questions of each type there are, and a second Counter tallies only the ones answered wrongly. Dividing one by the other, type by type, turns a vague sense that "it sometimes fails" into a list of where it fails.

Here, the spread-out and multi-hop questions fail most, which suggests trying multi-query retrieval and an agentic approach before anything else.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close