recallA retrieval
field guide

Error Analysis

Build this skill

Run error analysis with a fixed taxonomy, count failure categories and fix the largest first.

From individual failures to patterns

Debugging one wrong answer at a time is slow, and it can mislead: you fix whatever failure you happened to look at, which may be rare, while a common problem goes unnoticed. Error analysis is the practice of collecting many failures, sorting them into categories, counting them, and fixing the biggest categories first. It is one of the most effective habits in building any AI system, and it needs no special tools, just a spreadsheet and some patience.

The process

Collectfailurestest set, userreportsRead each onequestion,trace, answerLabel thecauseone maincategoryCountwhichcategoriesdominateFix thebiggestthen measureagain

Read failures, label their causes, count, and fix the largest category first.

  1. Collect 50 to 100 failures: wrong answers from the evaluation set, thumbs-down answers from users, and questions where the system said "I don't know" but the answer exists.
  2. Read each one fully: the question, the retrieved chunks with their ranks, the prompt and the answer.
  3. Label the main cause with one category from a fixed list (a taxonomy). Start with a simple list and add categories only when many failures don't fit.
  4. Count the categories.
  5. Fix the largest category, re-run the tests, and repeat.

A taxonomy for RAG errors

A practical starting taxonomy groups errors by the stage where the answer was lost:

Category Meaning
Content gap The documents do not contain the answer
Ingestion error Parsing or OCR damaged the evidence
Chunking error The answer was split or lost its context
Missed retrieval The right chunk was not found at all
Ranking error The right chunk was found but ranked below K
False retrieval Wrong or outdated chunks were retrieved and used
Generation error The evidence was in the prompt but the answer was wrong
Citation error The answer was right but cited the wrong source
Question problem The question was ambiguous or out of scope

Some categories have a clear owner. Content gaps belong to the people who write the documents, not the engineers: the fix is to write the missing policy or FAQ. Making that visible is one of error analysis's most useful outcomes.

Counting what matters

The program below counts labelled failures and shows what share each category accounts for, which tells you where to focus.

python · runs in your browser

Expected output:

text
missed retrieval    14    28%   cumulative   28%
ranking error       11    22%   cumulative   50%
content gap          9    18%   cumulative   68%
generation error     6    12%   cumulative   80%
chunking error       5    10%   cumulative   90%
false retrieval      3     6%   cumulative   96%
citation error       2     4%   cumulative  100%

counts.most_common() returns the categories from the most frequent to the least. running adds each count to the ones before it, and dividing by the total gives the cumulative share, which shows how few categories account for most of the failures.

Three categories (missed retrieval, ranking and content gaps) account for more than two-thirds of the failures. Better retrieval (hybrid search, query rewriting), a reranker, and a list of missing documents for the content owners would address most of the problems; tuning the prompt would address only about one failure in eight.

Practise labelling

Read each failure below and label its main cause.

Making error analysis a habit

Error analysis is most valuable when it is repeated regularly, for example every two weeks or after every major change, because the mix of errors shifts as you fix things. Keep the labelled examples: they become new test items, and the counts over time show whether the system is really getting better. And share the results beyond the engineering team. A chart showing that one in six failures is a content gap is a strong argument for the hostel office to finally write down its visitor policy.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close