Ragas Metrics Explained
with Bad Pizza & Worse Dates
Context Relevance · Answer Relevancy · Faithfulness · Answer Correctness · Recall@K · MRR · NDCG — all the numbers that tell you whether your RAG is a genius or just confidently wrong.
Why bother measuring?
Your RAG system can fail in two completely different ways:
It hands the LLM a pile of irrelevant documents (or misses the good ones). The model then either hallucinates or says “I don’t know” while looking guilty.
It gets perfect context… and still invents facts, wanders off-topic, or answers a different question entirely.
End-to-end “accuracy” hides which half is broken. Ragas splits the problem so you can fix the right thing.
The metric map
| Metric | Asks | Needs ground truth? |
|---|---|---|
| Faithfulness | Does every claim in the answer come from the context? | No |
| Answer Relevancy | Does the answer actually address the question? | No |
| Context Precision / Relevance | Are the retrieved chunks useful and ranked well? | Optional |
| Context Recall | Did we retrieve everything needed to answer? | Yes |
| Answer Correctness | How close is the answer to the gold answer? | Yes |
| Recall@K · MRR · NDCG | Classic ranking quality of the retriever | Yes (relevance labels) |
Faithfulness “Did you make that up?”
Score = (claims supported by context) / (total claims in the answer)
Retrieved context: “Our thin-crust pizza is ready in 12–15 minutes.”
✅ Faithful answer: “It takes about 12–15 minutes.”
❌ Unfaithful answer: “It takes 8 minutes and comes with free ice cream.”
The second answer invented two facts. Faithfulness ≈ 0.
Answer Relevancy “Did you even listen to the question?”
Ragas generates questions from the answer, then measures how similar those questions are to the original one.
✅ Relevant: “Click ‘Forgot password’, enter your email, and follow the link.”
❌ Irrelevant: “Passwords are very important for security. Always use strong ones.”
Technically true… but completely useless. Relevancy tanks.
Context Relevance / Precision & Recall
These measure the retriever, not the generator.
Context Precision (Relevance)
Of the chunks you retrieved, how many were actually useful — and were the good ones ranked near the top?
You asked for “refund policy for digital products.”
Top results: refund docs → high precision
Top results: company history, office locations, CEO bio → low precision
Context Recall
Did the retriever find all the information needed to answer the question?
Gold answer needs both “14-day window” and “order ID required.”
Both facts present in retrieved chunks → high recall
Only one fact retrieved → low recall
Answer Correctness “How close to the gold answer?”
Combines factual overlap (F1 of statements) + semantic similarity. Needs a ground-truth answer.
✅ High correctness: “In 1879, Einstein was born in Germany.”
❌ Low correctness: “Einstein was born in Spain in 1879.”
Recall@K · MRR · NDCG Classic retrieval ranking
These come from information retrieval and are often used alongside Ragas to judge how well the retriever ranks documents.
Recall@K
Out of all relevant documents, what fraction appear in the top-K results?
3 relevant docs exist. Top-5 contains 2 of them → Recall@5 = 2/3 ≈ 0.67
Funny version: “Did the search results contain the answers you needed within the first K?”
MRR (Mean Reciprocal Rank)
Looks at the rank of the first relevant document. Score = 1/rank.
First relevant doc is at position 1 → MRR = 1.0
Position 3 → MRR = 0.33
Not in top results → 0
“How high was the very first useful result?”
NDCG
Normalized Discounted Cumulative Gain. Rewards relevant docs higher in the ranking and penalizes putting them lower.
Perfect ranking of relevant docs → NDCG = 1.0
Relevant docs buried at the bottom → lower score
“Did we put the best stuff first?”
Sample Ragas code
Minimal examples using the modern Ragas style. Install with pip install ragas datasets openai.
1. Faithfulness + Answer Relevancy (no ground truth needed)
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy
data = {
"question": [
"How long does the thin-crust pizza take?",
"How do I reset my password?"
],
"answer": [
"It is ready in 12–15 minutes.",
"Passwords are very important for security."
],
"contexts": [
["Our thin-crust pizza is ready in 12–15 minutes."],
["Password resets start from the login page via Forgot password."]
]
}
dataset = Dataset.from_dict(data)
result = evaluate(
dataset,
metrics=[faithfulness, answer_relevancy]
)
print(result)
print(result.to_pandas())
2. With ground truth (Context Recall + Answer Correctness)
from ragas.metrics import context_recall, answer_correctness
data = {
"question": ["When was the first Super Bowl?"],
"answer": ["The first Super Bowl was held on Jan 15, 1967"],
"contexts": [[
"The First AFL–NFL World Championship Game was played on January 15, 1967."
]],
"ground_truth": ["The first Super Bowl was held on January 15, 1967"]
}
dataset = Dataset.from_dict(data)
result = evaluate(
dataset,
metrics=[context_recall, answer_correctness]
)
print(result)
3. Single-sample async style (newer API)
from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import Faithfulness
client = AsyncOpenAI()
llm = llm_factory("gpt-4o-mini", client=client)
scorer = Faithfulness(llm=llm)
result = await scorer.ascore(
user_input="When was the first Super Bowl?",
response="The first Super Bowl was held on Jan 15, 1967",
retrieved_contexts=[
"The First AFL–NFL World Championship Game was played on January 15, 1967."
]
)
print(f"Faithfulness: {result.value}")
Flashcards — test yourself
Quick mental models
- Faithfulness → “No made-up facts”
- Answer Relevancy → “Actually answered the question”
- Context Precision → “Useful chunks, ranked high”
- Context Recall → “Didn’t miss needed info”
- Answer Correctness → “Close to the gold answer”
- Recall@K → “Relevant docs in top-K?”
- MRR → “How high is the first good hit?”
- NDCG → “Best stuff ranked first?”
No comments:
Post a Comment