Evaluating Generated Answers
Measure faithfulness, answer relevance, context precision and context recall, and use LLM judges with care.
Retrieval was fine. Was the answer?
Good retrieval is necessary but not enough. Even with the right passages in the prompt, the model can ignore them, misread a number, add an invented detail, or answer a slightly different question. Evaluating the generated answer needs its own metrics.
Because there is rarely one exact correct wording, answers cannot be checked by simple string matching. Instead, answer quality is broken down into a few properties, each of which can be checked separately.
Four properties of a good RAG answer
Faithfulness (also called groundedness): is every claim in the answer supported by the retrieved passages? An answer that says "12 days" when the sources say "10 days", or adds a fee the sources never mention, is unfaithful. This is the most important RAG-specific metric, because it measures hallucination relative to the evidence.
Answer relevance: does the answer actually address the question that was asked, completely and without wandering off? A faithful answer about attendance rules is still a poor answer to a question about leave limits.
Context precision: of the passages given to the model, how many were actually relevant? Low precision means the prompt was full of noise.
Context recall: did the retrieved passages contain all the information needed to answer, judged against a reference answer? Low recall means the evidence was incomplete before generation even started.
Each metric compares two of the three things in a RAG run: the question, the retrieved context and the answer (plus a reference answer for recall).
Faithfulness and answer relevance can be measured without a reference answer, because they compare the answer with the context and the question. That makes them usable on live traffic, where there is no known correct answer.
Measuring faithfulness
The usual method is to split the answer into individual claims (short factual statements), then check each claim against the retrieved context. Faithfulness is the share of claims that the context supports.
"You can take up to 10 days of medical leave. Beyond that, the Dean of Academics must approve. Submitting the certificate late incurs a ₹200 fine."
With context saying only the first two things, the claims are: (1) up to 10 days — supported; (2) Dean approves beyond that — supported; (3) ₹200 fine for late certificates — not supported. Faithfulness = 2/3 ≈ 0.67.
Try being the judge. Flag each claim the sources do not support.
LLM-as-judge
Checking claims by hand does not scale to hundreds of questions after every change. The common solution is LLM-as-judge: ask a capable language model to do the judging, with a clear rubric. For faithfulness, the judge model is asked to list the claims in an answer and, for each, say whether the context supports it. For relevance, it is asked to rate how well the answer addresses the question.
LLM judges are fast and cheap compared with people, and they agree with human judgement reasonably well on clear criteria. They also have known weaknesses: they can favour longer answers, prefer answers in their own style, be swayed by the order in which options are shown, and miss subtle numerical errors. So their scores should be calibrated: have people judge a sample of answers, compare with the LLM judge, and only trust the judge for the criteria where they agree well.
The program below shows the scoring arithmetic once claims have been judged (by a person or a model), across several test answers.
Expected output:
answer 1: faithfulness 0.67
answer 2: faithfulness 1.00
answer 3: faithfulness 0.67
mean faithfulness: 0.78
answers with any unsupported claim: 2 of 3
For each answer, sum(ok for _, ok in claims) / len(claims) divides the number of supported claims by the number of claims. sum(s < 1 for s in scores) counts the answers whose score fell short of 1, meaning at least one claim was unsupported.
The last line is often more useful than the average: two of three answers contained at least one unsupported claim, which is what a user would actually experience.