Benchmarking Retrieval Configurations
Benchmark retrieval configurations fairly with Recall@K, MRR and ablations, reading per-question and per-type results.
Comparing settings fairly
A retrieval system has many settings: chunk size, embedding model, BM25 or vectors or both, top-K, reranker on or off. Each choice is a configuration. Benchmarking means running several configurations on the same evaluation dataset, measuring them the same way, and choosing based on the numbers rather than on impressions.
Retrieval should be benchmarked separately from generation. If the right passage never reaches the prompt, no model can produce a correct answer, and if you only score final answers you cannot tell a retrieval problem from a generation problem. Retrieval metrics also run fast and cost nothing, because they need no language model: you just check whether the known relevant chunks appear in the results.
Choosing the metrics
The standard ranking metrics all apply, and each answers a slightly different question.
| Metric | Question it answers for RAG |
|---|---|
| Hit rate @K | Did at least one relevant chunk make the top K? |
| Recall@K | What share of all relevant chunks made the top K? (Important for multi-part questions) |
| Precision@K | How much of the top K is relevant? (Noise in the prompt) |
| MRR | How high is the first relevant chunk? |
| NDCG@K | Are the most relevant chunks ranked highest? |
For RAG, Recall@K at the K you actually put in the prompt is usually the most important number: it measures whether the model receives the evidence it needs. Also look at a larger K (say 20): if Recall@20 is high but Recall@5 is low, the right chunks are being found but ranked too low, which points to a reranker.
Running a benchmark
The program below benchmarks three configurations on a small test set. The retrieved lists are written out by hand to keep it short; in practice they come from running each configuration.
Expected output:
configuration Recall@3 MRR
BM25 only 0.70 0.60
Vectors only 1.00 0.87
Hybrid + reranker 1.00 1.00
| Function | What it does |
|---|---|
recall(ranked, relevant) |
Takes the top K results, counts how many of the relevant chunks are among them, and divides by the number of relevant chunks. |
rr(ranked, relevant) |
Walks down the ranking and returns 1 divided by the position of the first relevant chunk, or 0 if there is none. Averaged over the questions, this is MRR. |
The loop runs both functions on every question for one configuration and averages them, so each row of the output summarises five questions.
Ablation: change one thing at a time
When a configuration differs from another in several ways, you cannot tell which change made the difference. An ablation study changes one component at a time, starting from a baseline: baseline → + hybrid search → + reranker → + contextual chunks. Each step's effect is then clear, and you can drop components that add cost without adding quality.
Adding one component at a time shows which ones pay for themselves. The last step adds little for a large indexing cost.
Comparing fairly
A few rules keep comparisons honest:
- Same test set, same version, for every configuration.
- Look at per-question results, not only averages. A configuration can raise the average while breaking an important group of questions, such as all the Hindi ones.
- Break results down by question type. Hybrid search may help code-like questions a lot and paraphrased ones a little.
- Beware small differences. With 50 questions, a 2-point difference is often just noise; prefer differences that are large, consistent across types, and repeatable.
- Measure cost and speed too. Record latency (average and the slowest 5 percent) and cost per query next to the quality numbers. A small gain may not be worth doubling the response time.