recallA retrieval
field guide

Regression Testing for RAG

Build this skill

Protect quality with regression tests, quality gates and checks that adapt when documents change.

Every change can break something

A RAG system changes constantly. New documents are added every day. The embedding model is upgraded. Someone tweaks the prompt to fix one complaint. The chunk size is changed. A library is updated. Each of these can quietly make some answers worse while making others better. A regression is something that used to work and no longer does, and regression testing means re-running a fixed set of tests after every change to catch regressions before users do.

For ordinary software, regression tests check exact outputs. For RAG, answers vary in wording and the documents themselves keep changing, so regression tests check metrics and properties instead: retrieval recall, faithfulness, whether key facts appear, and whether the system still refuses to answer what the documents don't cover.

What to run, and when

Changecode, model,prompt ordocumentsRun test setretrieval +answersComparewith the lastgood runGatepassthresholds?Release orblockwith a report

Every change triggers the same tests, compared against the last good run. A drop beyond the allowed margin blocks the change.

A good regression suite has a few layers:

  • Fast checks on every change: retrieval metrics (Recall@K, MRR) on the full test set, which need no language model and run in seconds or minutes.
  • Answer checks: faithfulness and correctness on the test set using an evaluation framework, run on every significant change or nightly.
  • Must-pass questions: a small list of critical questions (fees, deadlines, safety rules) whose answers must contain specific facts, and unanswerable questions that must be refused.
  • Document-update checks: when the knowledge base is refreshed, re-run the questions whose sources changed, and update reference answers that are now out of date.

Thresholds and gates

A quality gate is a rule that decides whether a change may go live. Gates usually combine two kinds of threshold:

  • Absolute minimums: "faithfulness must be at least 0.90", "Recall@5 must be at least 0.85".
  • Maximum drops: "no metric may fall more than 2 points below the last release", "no must-pass question may fail".

Allowing a small drop matters, because metrics wobble a little between runs (especially those judged by a language model). A gate that fails on every tiny wobble gets ignored.

The program below compares a new run with a baseline and applies a simple gate.

python · runs in your browser

Expected output:

text
recall@5               0.86 -> 0.88  (+0.02)  
mrr                    0.74 -> 0.75  (+0.01)  
faithfulness           0.93 -> 0.87  (-0.06)  below minimum 0.9
refuses_unanswerable   1.00 -> 1.00  (+0.00)  

GATE: BLOCKED: faithfulness

Each metric is tested against two rules. If it has an agreed minimum and the new value is below it, the gate fails, however small the drop. Otherwise the gate fails only if the metric fell by more than max_drop, which stops ordinary run-to-run noise from blocking a release.

Retrieval improved, but faithfulness fell below its minimum, so the change is blocked. Perhaps the new configuration retrieves more passages, and the extra text is leading the model astray. Without the gate, users would have seen more invented details while the team celebrated better recall.

When the documents change, not the code

RAG has a twist that ordinary software lacks: the data changes too. When the leave limit changes from 15 days to 10, the old reference answer becomes wrong, and a correct system will appear to "fail" that test. Two practices handle this:

  • Link each test item to its source documents, so that when a source changes, the affected test items are flagged for review.
  • Keep property-style checks where possible ("the answer cites the current regulations", "the answer contains the number from the source") rather than fixed expected answers, so tests adapt as documents change.

Make it automatic

Regression tests only help if they actually run. The usual approach is to add them to the team's continuous integration (CI) pipeline, the automated system that runs tests whenever code is changed, and to schedule a nightly run against the latest documents. Each run saves a short report (metrics, changes from the baseline, and the questions that changed from pass to fail), so that anyone can see what a change did.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close