Anatomy of a RAG System
Map every stage of a RAG system, from parsing to evaluation, and trace a wrong answer back to the stage that caused it.
What happens between the question and the answer
A student types "Can I take 12 days of medical leave, and who approves it?" into the Kaveri Institute's assistant. Two seconds later she sees a correct answer that cites Section 4 of the academic regulations. From the outside it looks like one step. Inside, it is two separate pipelines that run at different times, plus a way of checking the results.
The offline pipeline (also called the indexing pipeline) runs whenever documents are added or changed. It reads the institute's PDFs, web pages and circulars, cuts them into passages and stores the passages so they can be searched quickly. Nobody is waiting for it, so it can be slow and thorough.
The online pipeline (also called the query pipeline) runs every time someone asks a question. It finds the right passages, builds a prompt and writes the answer. Someone is waiting, so it has to be fast.
Around both sits an evaluation loop, which checks whether answers are right and, when they are not, helps you find which step to fix.
The offline pipeline: preparing the documents
Ahead of time: each document is read, split into passages, turned into numbers and stored with its details.
Reading and parsing
First, each file is opened and its text pulled out, keeping as much structure as possible: headings, lists, tables and page numbers. This is harder than it sounds. A PDF stores letters at positions on a page rather than paragraphs in reading order, a scanned circular is just a picture until a text-recognition program (OCR) reads it, and a table that loses its rows and columns turns into a jumble of numbers. Tools such as Docling and MarkItDown exist to convert many file types into clean, structured text.
Chunking
Next, each document is split into smaller passages called chunks, usually a few hundred words long. A chunk should be small enough that a search returns just the part that answers a question, but big enough to make sense on its own. A chunk that says only "above 10 · Dean of Academics" has lost the heading that explains what those words refer to.
Embedding
Each chunk is then turned into an embedding: a long list of numbers, produced by an embedding model such as BGE-M3, that represents the chunk's meaning. Chunks with similar meanings get similar lists of numbers. That is what lets the system find "medical leave" when someone asks about "sick days". Many systems also keep an ordinary keyword index, because exact matching still works best for course codes, form numbers and names.
Indexing
Finally, the numbers, the chunk text and its metadata (details such as the source document, section, date and who may see it) are stored in a database built for fast searching. PostgreSQL with the pgvector extension is a common choice. Metadata turns out to be essential: it is how the system cites a source, ignores last year's version, or keeps a staff-only circular away from students.
The online pipeline: answering a question
Every question: find chunks, keep the best, build the prompt, write a cited answer. Evaluation checks the result and points to the step to fix.
Search (retrieval)
The question is turned into numbers with the same embedding model, and the system finds the chunks whose numbers are closest, often together with a keyword search. This step is quick and casts a wide net, returning perhaps 20 to 50 candidates.
Reranking
A slower, more careful model reads the question and each candidate together and scores them again. Its job is to push the chunk that really answers the question to the top, so that the few places in the prompt go to the best evidence.
Building the prompt
The system puts together the instructions ("answer only from the sources below, cite them, and say so if they do not contain the answer"), the top few chunks, numbered so they can be cited, and the question.
Generation
The prompt goes to a language model, which writes the answer with citations. Frameworks such as LangChain provide ready-made building blocks for these steps, but the steps are the same whichever tool connects them.
| Step | Takes in | Gives out | Example tools |
|---|---|---|---|
| Read and parse | PDFs, Word files, web pages, scans | Clean text with headings and tables | Docling, MarkItDown, OCR |
| Chunk | Clean text | Passages a few hundred words long | Text splitters |
| Embed | Passages | One list of numbers per passage | BGE-M3 |
| Index | Numbers, text, metadata | A searchable store | PostgreSQL + pgvector |
| Search | The question | 20 to 50 candidate passages | Vector search, BM25 |
| Rerank | Question and candidates | The best few, in order | Reranking models |
| Build prompt | Best passages and question | The prompt | Prompt templates |
| Generate | The prompt | Answer with citations | Any capable language model |
| Evaluate | Logs of each run | Quality scores and traces | LangSmith, test question sets |
Checking the answers: the evaluation loop
A RAG system can go wrong in many places, and the final answer rarely tells you where. The evaluation loop makes problems visible.
Offline, you keep a fixed set of test questions with known answers and known source passages, and run them after every change. For each one you check two things: did the search find the right passage, and is every claim in the answer supported by the passages it was given?
Online, every real question is recorded as a trace: the question, the passages found and their scores, the exact prompt, the answer, how long it took and how many tokens it used. Tools such as LangSmith make these traces easy to search, and user feedback, like a thumbs-down, points to the traces worth reading first.
The point of all this is diagnosis. When an answer is wrong, the trace shows whether the right passage was never stored, was stored but not found, was found but dropped, or was in the prompt and ignored by the model. Each has a different fix in a different step.
Following a question, then breaking it
The lab below follows the medical leave question through six simplified stages. First step through it with nothing broken and look at what each stage produces. Then pick a broken scenario and step through again. The broken stage is marked in red, and you can watch the problem travel all the way to the final answer. Notice which broken answers still look perfectly reasonable.
Finding the broken stage
The lab shows a pattern worth remembering: a problem in one stage usually shows up somewhere else, most often in the final answer. Teams that only look at final answers end up rewriting prompts to fix what are really file-reading or storage problems.
Practise reading the symptoms. For each one below, pick the stage you would check first.
Three habits help. Look at what each stage produces, not just the final answer: read a sample of extracted text and chunks before trusting any score. Test the search separately from the answer writing, because a perfect model cannot use a passage it never received, and a perfect search cannot stop a model from ignoring its evidence. And change one stage at a time, re-running the same test questions after each change, so you always know what caused an improvement or a new problem.