recallA retrieval
field guide

Anatomy of a RAG System

Build this skill

Map every stage of a RAG system, from parsing to evaluation, and trace a wrong answer back to the stage that caused it.

What happens between the question and the answer

A student types "Can I take 12 days of medical leave, and who approves it?" into the Kaveri Institute's assistant. Two seconds later she sees a correct answer that cites Section 4 of the academic regulations. From the outside it looks like one step. Inside, it is two separate pipelines that run at different times, plus a way of checking the results.

The offline pipeline (also called the indexing pipeline) runs whenever documents are added or changed. It reads the institute's PDFs, web pages and circulars, cuts them into passages and stores the passages so they can be searched quickly. Nobody is waiting for it, so it can be slow and thorough.

The online pipeline (also called the query pipeline) runs every time someone asks a question. It finds the right passages, builds a prompt and writes the answer. Someone is waiting, so it has to be fast.

Around both sits an evaluation loop, which checks whether answers are right and, when they are not, helps you find which step to fix.

The offline pipeline: preparing the documents

Read filespull out the textChunksplit into passagesEmbedturn text intonumbersIndexstore for fastsearch

Ahead of time: each document is read, split into passages, turned into numbers and stored with its details.

Reading and parsing

First, each file is opened and its text pulled out, keeping as much structure as possible: headings, lists, tables and page numbers. This is harder than it sounds. A PDF stores letters at positions on a page rather than paragraphs in reading order, a scanned circular is just a picture until a text-recognition program (OCR) reads it, and a table that loses its rows and columns turns into a jumble of numbers. Tools such as Docling and MarkItDown exist to convert many file types into clean, structured text.

Chunking

Next, each document is split into smaller passages called chunks, usually a few hundred words long. A chunk should be small enough that a search returns just the part that answers a question, but big enough to make sense on its own. A chunk that says only "above 10 · Dean of Academics" has lost the heading that explains what those words refer to.

Embedding

Each chunk is then turned into an embedding: a long list of numbers, produced by an embedding model such as BGE-M3, that represents the chunk's meaning. Chunks with similar meanings get similar lists of numbers. That is what lets the system find "medical leave" when someone asks about "sick days". Many systems also keep an ordinary keyword index, because exact matching still works best for course codes, form numbers and names.

Indexing

Finally, the numbers, the chunk text and its metadata (details such as the source document, section, date and who may see it) are stored in a database built for fast searching. PostgreSQL with the pgvector extension is a common choice. Metadata turns out to be essential: it is how the system cites a source, ignores last year's version, or keeps a staff-only circular away from students.

The online pipeline: answering a question

Searchfind candidatechunksRerankpick the bestfewBuild promptinstructions +passagesGeneratewrite and citeEvaluatelog and checkproblems are traced back to the step that caused them

Every question: find chunks, keep the best, build the prompt, write a cited answer. Evaluation checks the result and points to the step to fix.

Search (retrieval)

The question is turned into numbers with the same embedding model, and the system finds the chunks whose numbers are closest, often together with a keyword search. This step is quick and casts a wide net, returning perhaps 20 to 50 candidates.

Reranking

A slower, more careful model reads the question and each candidate together and scores them again. Its job is to push the chunk that really answers the question to the top, so that the few places in the prompt go to the best evidence.

Building the prompt

The system puts together the instructions ("answer only from the sources below, cite them, and say so if they do not contain the answer"), the top few chunks, numbered so they can be cited, and the question.

Generation

The prompt goes to a language model, which writes the answer with citations. Frameworks such as LangChain provide ready-made building blocks for these steps, but the steps are the same whichever tool connects them.

Step Takes in Gives out Example tools
Read and parse PDFs, Word files, web pages, scans Clean text with headings and tables Docling, MarkItDown, OCR
Chunk Clean text Passages a few hundred words long Text splitters
Embed Passages One list of numbers per passage BGE-M3
Index Numbers, text, metadata A searchable store PostgreSQL + pgvector
Search The question 20 to 50 candidate passages Vector search, BM25
Rerank Question and candidates The best few, in order Reranking models
Build prompt Best passages and question The prompt Prompt templates
Generate The prompt Answer with citations Any capable language model
Evaluate Logs of each run Quality scores and traces LangSmith, test question sets

Checking the answers: the evaluation loop

A RAG system can go wrong in many places, and the final answer rarely tells you where. The evaluation loop makes problems visible.

Offline, you keep a fixed set of test questions with known answers and known source passages, and run them after every change. For each one you check two things: did the search find the right passage, and is every claim in the answer supported by the passages it was given?

Online, every real question is recorded as a trace: the question, the passages found and their scores, the exact prompt, the answer, how long it took and how many tokens it used. Tools such as LangSmith make these traces easy to search, and user feedback, like a thumbs-down, points to the traces worth reading first.

The point of all this is diagnosis. When an answer is wrong, the trace shows whether the right passage was never stored, was stored but not found, was found but dropped, or was in the prompt and ignored by the model. Each has a different fix in a different step.

Following a question, then breaking it

The lab below follows the medical leave question through six simplified stages. First step through it with nothing broken and look at what each stage produces. Then pick a broken scenario and step through again. The broken stage is marked in red, and you can watch the problem travel all the way to the final answer. Notice which broken answers still look perfectly reasonable.

Finding the broken stage

The lab shows a pattern worth remembering: a problem in one stage usually shows up somewhere else, most often in the final answer. Teams that only look at final answers end up rewriting prompts to fix what are really file-reading or storage problems.

Practise reading the symptoms. For each one below, pick the stage you would check first.

Three habits help. Look at what each stage produces, not just the final answer: read a sample of extracted text and chunks before trusting any score. Test the search separately from the answer writing, because a perfect model cannot use a passage it never received, and a perfect search cannot stop a model from ignoring its evidence. And change one stage at a time, re-running the same test questions after each change, so you always know what caused an improvement or a new problem.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close