recallA retrieval
field guide

Building an Evaluation Dataset

Build this skill

Build a versioned evaluation dataset of realistic, answerable and unanswerable questions with reference answers and sources.

You cannot improve what you do not measure

Every change to a RAG system, whether a new chunk size, embedding model, reranker or prompt, makes some answers better and possibly others worse. Without a fixed set of test questions with known answers, you cannot tell which. Teams end up making decisions from a few memorable examples, and quality drifts without anyone noticing.

An evaluation dataset (also called a golden dataset or test set) is a collection of questions, each with the correct answer and the source passages that support it. It is the single most valuable asset for improving a RAG system, and building it is mostly careful, unglamorous work.

What each test item contains

Field Example Used to measure
Question "Can I take 12 days of medical leave?" Everything
Reference answer "Up to 10 days; more needs the Dean of Academics." Answer correctness
Relevant sources Regulations 2026 §4.1, §4.2 Retrieval (Recall@K, MRR)
Question type two-part / single fact / multi-hop Results broken down by type
Tags leave, students, English Finding weak areas

Not every item needs every field, but relevant sources are especially valuable: they let you measure retrieval separately from generation, which is essential for diagnosis.

Where good questions come from

The best test questions look like what real users ask, not like tidy textbook questions.

  • Real logs. Questions from search logs, helpdesk emails or an early pilot are the most realistic, including their typos, vagueness and code-mixing.
  • Domain experts. People who know the documents (the exam cell, the hostel office) can write questions they are asked every week, and the correct answers.
  • Synthetic generation. A language model can read chunks and write questions that each chunk answers. This quickly produces many items, but they tend to copy the document's wording and be easier than real questions, so they must be reviewed and should never be the whole set.
Collectlogs, experts,syntheticAnswerreferenceanswersLink sourceswhich chunkssupport itReviewremoveduplicates, fixerrorsVersionfreeze andtrack changes

Collect real questions, add answers and sources, review them, and keep the set under version control.

Covering the right mix

A test set should cover the kinds of questions the system will face, roughly in proportion, plus a few of the hard and risky ones even if they are rare:

  • simple single-fact questions;
  • questions phrased very differently from the documents;
  • multi-part and multi-hop questions;
  • questions about numbers, dates and versions;
  • questions in each language users actually use;
  • unanswerable questions, whose correct response is "the documents do not say this" (these test whether the system admits ignorance instead of inventing).

Fifty to a few hundred items is a practical size for most projects. Small sets are noisy: with 30 questions, one question changes accuracy by more than 3 percentage points.

The program below checks a small test set for balance across types and flags items that are missing sources.

python · runs in your browser

Expected output:

text
by type: {'single fact': 2, 'paraphrase': 1, 'multi-part': 1, 'multilingual': 1, 'unanswerable': 1}
missing sources: Hostel fee for 2026?

Counter tallies the items by type, which shows at a glance whether one type dominates the set. The loop then looks for items with an empty sources list. That is correct for an unanswerable question and a mistake for any other, since without sources nobody can score whether retrieval found the right passage.

Keeping the dataset honest

A few rules keep a test set trustworthy over time:

  • Freeze it and version it. Changing the questions while comparing systems makes results meaningless. When you add or fix items, record a new version.
  • Keep a held-out part. If you tune settings on the same questions you report, you will slowly overfit to them. Keep some items aside for final checks only.
  • Update it when documents change. If the leave limit changes from 15 to 10 days, the reference answers must change too, or the test set will punish the correct new answer.
  • Grow it from failures. Every real failure reported by a user is a perfect new test item.

Judge the quality of these candidate test items.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close