recallA retrieval
field guide

Evaluation Frameworks

Build this skill

Evaluate RAG with RAGAS and DeepEval, and calibrate their judge-based scores against human review.

Don't build every metric yourself

Faithfulness, answer relevance, context precision and context recall can all be implemented by hand, but doing so well takes time: writing judge prompts, splitting answers into claims, handling errors, aggregating scores. Several open-source evaluation frameworks package these metrics, so you can run them on your test set with a few lines of code and focus on interpreting the results. Two widely used ones are RAGAS and DeepEval.

RAGAS

RAGAS (Retrieval-Augmented Generation Assessment) began as a 2023 research paper by Shahul Es and colleagues that proposed reference-free evaluation of RAG: metrics such as faithfulness and answer relevance that can be computed from just the question, the retrieved contexts and the answer, using a language model as the judge, without needing a human-written correct answer for every question. It has since grown into a library with many metrics, including context precision and context recall (which do use a reference answer), and tools for generating synthetic test sets.

A RAGAS evaluation takes a dataset of samples, each holding the question, the retrieved contexts, the system's answer and (optionally) a reference answer, and returns a score per metric per sample. The example below shows the shape of the code. It runs on your own computer with a configured judge model; the library's interface has changed between versions, so check the current documentation for exact names.

python
# pip install ragas
from ragas import evaluate, EvaluationDataset
from ragas.metrics import Faithfulness, ResponseRelevancy, LLMContextRecall

samples = [{
    "user_input": "Can I take 12 days of medical leave?",
    "retrieved_contexts": ["Students may take up to 10 days of medical leave in a semester.",
                           "More than 10 days needs approval from the Dean of Academics."],
    "response": "You can take up to 10 days; the extra 2 days need the Dean of Academics' approval.",
    "reference": "Up to 10 days are allowed; more than 10 needs the Dean of Academics.",
}]
result = evaluate(dataset=EvaluationDataset.from_list(samples),
                  metrics=[Faithfulness(), ResponseRelevancy(), LLMContextRecall()],
                  llm=judge_llm, embeddings=judge_embeddings)
print(result)

DeepEval

DeepEval, from Confident AI, takes a testing-first approach modelled on unit tests in software development. Each question becomes a test case, each metric has a threshold, and a test passes or fails, so evaluations can run in the same automated pipelines that run ordinary software tests. It includes RAG metrics (faithfulness, answer relevancy, contextual precision, recall and relevancy) as well as general ones, and supports custom metrics defined in plain language.

python
# pip install deepeval
from deepeval import evaluate
from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric

case = LLMTestCase(
    input="Can I take 12 days of medical leave?",
    actual_output="You can take up to 10 days; the extra 2 days need the Dean's approval.",
    retrieval_context=["Students may take up to 10 days of medical leave in a semester.",
                       "More than 10 days needs approval from the Dean of Academics."],
)
evaluate(test_cases=[case], metrics=[FaithfulnessMetric(threshold=0.8), AnswerRelevancyMetric(threshold=0.7)])
RAGASResearch origin: reference-free RAG metricsScores per metric per sampleGood for analysis and comparisonTest set generation toolsDeepEvalTesting-first: cases, thresholds, pass/failFits automated test pipelinesRAG and general metricsCustom metrics in plain language

Both use language models as judges for RAG metrics. They differ mainly in workflow: analysis versus pass/fail testing.

Can you trust the scores?

Framework metrics are language-model judgements, and they inherit the strengths and weaknesses of LLM-as-judge. Before relying on them to make decisions, check them.

Calibrate against people

Take 30 to 50 answers, have someone who knows the domain mark each one (for example faithful or not), and compare with the framework's scores. If they mostly agree, the metric is useful for your data. If not, adjust the judge model, the thresholds or the metric.

Use a strong judge, and keep it fixed

A weak judge model gives noisy scores. Changing the judge model between runs changes the scores even if your system did not change, so record which judge was used.

Expect some randomness

Language model judges can give slightly different scores on repeated runs. Average over enough questions, and treat small differences with caution.

Watch the cost

Each metric may make several judge calls per question. A few hundred questions times four metrics can be thousands of calls.

The program below shows a simple calibration check: comparing judge decisions with human decisions on the same answers and computing the agreement rate.

python · runs in your browser

Expected output:

text
agreement: 10/12 = 83%
unfaithful answers the judge let through: 1
good answers the judge flagged wrongly:   1

zip(human, judge) pairs the two verdicts for each answer. The three sum(...) lines then count the pairs that agree, the pairs where a person said unfaithful and the judge said faithful, and the reverse.

Agreement of 83 percent is useful, but the one unfaithful answer that the judge let through is the kind of miss that matters most. A judge gives each answer a score, and the passing threshold decides which mistakes you get. Move it in the lab.

No threshold gets all eight right, because the judge gave a wrong answer a higher score than two right ones. The choice is which mistake to accept. If missed problems are costly, raise the passing mark and review what it flags, or use a stronger judge.

Choosing and using a framework

Frameworks make evaluation fast and repeatable, which is what allows it to happen after every change. They do not remove the need to read answers yourself. The most reliable teams combine automated metrics on every change with regular human review of a sample, especially of the failures the metrics flag.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close