Measuring Retrieval Quality
Measure a ranking with Precision@K, Recall@K, MRR and NDCG, and pick the metric that matches how results are used.
How do you know a search got better?
Suppose you switch the Kaveri Institute's search from TF-IDF to BM25. You try five queries and the results look better. Then a friend tries five other queries and finds two that look worse. Who is right? Without a proper way to measure, every change turns into an argument about examples.
Search quality can be measured, and the method is simple:
- Collect a set of test queries: real questions people ask.
- For each query, have a person mark which pages actually answer it. These marks are called relevance judgements.
- Run the search system on every test query and compare its results with the judgements, using a few standard scores.
Once the test set exists, you can re-run it after every change and see whether things improved. This recipe goes back to library experiments in the 1960s, and it is still how search systems are compared today, including at TREC, a yearly search competition run by the US National Institute of Standards and Technology since 1992.
Writing the answer key
The test queries and judgements together work like an answer key for an exam. A few practical points make a big difference.
Use queries that look like what people really type, including short, vague and misspelt ones, not tidy textbook questions. Fifty to a hundred queries is a reasonable start for a small system.
Judgements can be binary (relevant or not) or graded (for example 0 = not relevant, 1 = partly relevant, 2 = relevant, 3 = perfect answer). Graded judgements capture the difference between the Medical Leave policy, which answers Meera's question directly, and the Attendance Rules page, which only mentions medical leave in passing.
The person judging should know the material well, because "relevant" means "actually helps with the question", not "contains the right words".
Precision and recall
Most people only look at the first few results, and an AI assistant usually reads only the top few pages. So scores are measured at a cut-off called K, meaning "only count the top K results".
Imagine a search returns these top 5 results for "medical leave rules", and the collection holds 4 relevant pages in total.
Precision looks at the left column: how much of it is useful. Recall looks at the right column: how much of it was found.
Precision@K asks: of the top K results, what fraction are relevant? Here 3 of the 5 are, so Precision@5 = 3 ÷ 5 = 0.6.
Recall@K asks: of all the relevant pages that exist, what fraction made it into the top K? Here 3 of the 4 did, so Recall@5 = 3 ÷ 4 = 0.75.
The two often pull in opposite directions. Showing more results usually raises recall (you find more of the good pages) but lowers precision (more junk sneaks in). Which one matters more depends on the job. A student looking for one rule wants high precision at the top. A lawyer checking every past case needs high recall. For an AI assistant, recall at a small K matters most: if the page with the answer is not among the few pages the model reads, the model cannot use it.
Reciprocal rank: how soon is the first good result?
Precision@5 gives the same score whether the good result is at #1 or at #5. For a person scanning results, that difference matters a lot. Reciprocal rank looks only at where the first relevant result appears: 1 if it is first, 1/2 if second, 1/3 if third, and 0 if none appears.
| Query | First relevant result at | Reciprocal rank |
|---|---|---|
| medical leave rules | #2 | 1/2 = 0.50 |
| hostel fees | #1 | 1/1 = 1.00 |
| bonafide certificate | #4 | 1/4 = 0.25 |
| Average over all queries (MRR) | 0.58 |
The average across all test queries is called Mean Reciprocal Rank (MRR). It is a good fit when each query has one right answer, such as finding a particular form.
Try the scores yourself. Mark each result as relevant or not, move results up, and watch the three numbers change. Before each click, guess which numbers will move.
NDCG: better pages should be higher up
The most widely used ranking score combines two ideas: a perfect answer is worth more than a partly useful one, and a good page near the top is worth more than the same page further down. It is called NDCG (Normalised Discounted Cumulative Gain), and the name describes how it works.
- Gain: each result earns points for how relevant it is. A common choice is 2^grade − 1, so a perfect answer (grade 3) earns 7 points and a partly relevant one (grade 1) earns 1 point.
- Discounted: points shrink the further down the list a result appears. A result at #2 keeps about 63% of its points, at #3 half, and at #10 less than a third.
- Cumulative: add up the discounted points for the top K results. This total is the DCG.
- Normalised: divide by the best possible DCG, the one you would get if the results were in the perfect order. This makes NDCG a number between 0 and 1, where 1 means "perfect order".
The discount: the same perfect answer is worth much less at #10 than at #1.
Worked example: NDCG for five results
The five results have grades 0, 3, 0, 2, 1, so they earn 0, 7, 0, 3 and 1 points. Dividing by the discounts for positions 1 to 5 (1, 1.58, 2, 2.32, 2.58) gives 0 + 4.42 + 0 + 1.29 + 0.39 = 6.10. That is the DCG.
The perfect order for this query (which also has one grade-2 page the search missed) would be grades 3, 2, 2, 1, 0, worth 7 + 1.89 + 1.5 + 0.43 = 10.82.
NDCG = 6.10 ÷ 10.82 = 0.56. If the grade-3 page moved from #2 to #1, NDCG would rise to 0.80, while precision and recall would not change at all. That is exactly what NDCG is designed to notice.
The scores in code
Each score takes only a few lines of Python. The program below scores the five results from the example. Change the list of True/False values and run it again.
Expected output:
Precision@5: 0.6
Recall@5: 0.75
Reciprocal rank: 0.5
sum(results) counts the relevant results, because Python treats True as 1 when adding. results.index(True) gives the position of the first relevant result counting from 0, so 1 is added to turn it into a rank.
Picking the right score
No single number tells the whole story, so teams usually track two or three.
| Score | Question it answers | Good for |
|---|---|---|
| Precision@K | How much of what we showed was useful? | Result pages with little space |
| Recall@K | How much of what exists did we find? | Feeding pages to an AI model; legal search |
| MRR | How soon does the first good result appear? | Finding one specific page or form |
| NDCG@K | Are the best pages in the best positions? | Overall ranking quality |
When comparing two systems, run them on exactly the same queries and judgements, and look at individual queries as well as averages. An average that improves slightly can hide a few queries that got much worse. And keep some test queries aside that you never tune on; otherwise you end up with a system that is very good at those particular queries and not much else.
Test yourself on matching the score to the job.