Saturday, September 19, 2026

Ragas Metrics — The Funny Guide to Evaluating RAG
A slightly unhinged guide

Ragas Metrics Explained
with Bad Pizza & Worse Dates

Context Relevance · Answer Relevancy · Faithfulness · Answer Correctness · Recall@K · MRR · NDCG — all the numbers that tell you whether your RAG is a genius or just confidently wrong.

LLM-as-judge No more “looks fine to me” Sample code included

Why bother measuring?

Your RAG system can fail in two completely different ways:

Retriever fails
It hands the LLM a pile of irrelevant documents (or misses the good ones). The model then either hallucinates or says “I don’t know” while looking guilty.
Generator fails
It gets perfect context… and still invents facts, wanders off-topic, or answers a different question entirely.

End-to-end “accuracy” hides which half is broken. Ragas splits the problem so you can fix the right thing.

The metric map

flowchart TB subgraph RET["Retrieval Quality"] CP["Context Precision / Relevance"] CR["Context Recall"] RK["Recall@K"] MRR_N["MRR"] NDCG_N["NDCG"] end subgraph GEN["Generation Quality"] F["Faithfulness"] AR["Answer Relevancy"] AC["Answer Correctness"] end Q["User Question"] --> RET RET --> CTX["Retrieved Context"] CTX --> GEN GEN --> A["Final Answer"] GT["Ground Truth"] -.-> AC GT -.-> CR
MetricAsksNeeds ground truth?
FaithfulnessDoes every claim in the answer come from the context?No
Answer RelevancyDoes the answer actually address the question?No
Context Precision / RelevanceAre the retrieved chunks useful and ranked well?Optional
Context RecallDid we retrieve everything needed to answer?Yes
Answer CorrectnessHow close is the answer to the gold answer?Yes
Recall@K · MRR · NDCGClassic ranking quality of the retrieverYes (relevance labels)

Faithfulness “Did you make that up?”

Score = (claims supported by context) / (total claims in the answer)

Question: “How long does the pizza take?”

Retrieved context: “Our thin-crust pizza is ready in 12–15 minutes.”

✅ Faithful answer: “It takes about 12–15 minutes.”

❌ Unfaithful answer: “It takes 8 minutes and comes with free ice cream.”

The second answer invented two facts. Faithfulness ≈ 0.

Think of it like a friend retelling a story. If they add “and then a dragon appeared,” you know they hallucinated — even if the rest is true.
flowchart LR A["Generated Answer"] --> S["Split into atomic claims"] S --> C1["Claim 1"] S --> C2["Claim 2"] S --> C3["Claim 3"] C1 --> J{"Supported by context?"} C2 --> J C3 --> J J -->|Yes| Y["Yes"] J -->|No| N["No"] Y --> SCORE["Faithfulness = supported / total"] N --> SCORE

Answer Relevancy “Did you even listen to the question?”

Ragas generates questions from the answer, then measures how similar those questions are to the original one.

Question: “How do I reset my password?”

✅ Relevant: “Click ‘Forgot password’, enter your email, and follow the link.”

❌ Irrelevant: “Passwords are very important for security. Always use strong ones.”

Technically true… but completely useless. Relevancy tanks.

Real-life analogy: You ask “What time is the movie?” and your date replies “I love cinema, the popcorn here is great, and the seats are comfy.” Faithful to the theater experience, zero relevancy to the clock.

Context Relevance / Precision & Recall

These measure the retriever, not the generator.

Context Precision (Relevance)

Of the chunks you retrieved, how many were actually useful — and were the good ones ranked near the top?

You asked for “refund policy for digital products.”

Top results: refund docs → high precision

Top results: company history, office locations, CEO bio → low precision

Context Recall

Did the retriever find all the information needed to answer the question?

Gold answer needs both “14-day window” and “order ID required.”

Both facts present in retrieved chunks → high recall

Only one fact retrieved → low recall

Analogy: You’re making a sandwich. Precision = “most of the ingredients you grabbed are for the sandwich.” Recall = “you didn’t forget the bread.”

Answer Correctness “How close to the gold answer?”

Combines factual overlap (F1 of statements) + semantic similarity. Needs a ground-truth answer.

Ground truth: “Einstein was born in 1879 in Germany.”

✅ High correctness: “In 1879, Einstein was born in Germany.”

❌ Low correctness: “Einstein was born in Spain in 1879.”

Think of it as grading a student essay against the answer key — both facts and meaning matter.

Recall@K · MRR · NDCG Classic retrieval ranking

These come from information retrieval and are often used alongside Ragas to judge how well the retriever ranks documents.

Recall@K

Out of all relevant documents, what fraction appear in the top-K results?

3 relevant docs exist. Top-5 contains 2 of them → Recall@5 = 2/3 ≈ 0.67

Funny version: “Did the search results contain the answers you needed within the first K?”

MRR (Mean Reciprocal Rank)

Looks at the rank of the first relevant document. Score = 1/rank.

First relevant doc is at position 1 → MRR = 1.0
Position 3 → MRR = 0.33
Not in top results → 0

“How high was the very first useful result?”

NDCG

Normalized Discounted Cumulative Gain. Rewards relevant docs higher in the ranking and penalizes putting them lower.

Perfect ranking of relevant docs → NDCG = 1.0
Relevant docs buried at the bottom → lower score

“Did we put the best stuff first?”

flowchart LR Q["Query"] --> R["Retriever"] R --> L["Ranked list: D1, D2, D3, D4, D5"] L --> REC["Recall@K"] L --> MRR_M["MRR"] L --> NDC["NDCG"]

Sample Ragas code

Minimal examples using the modern Ragas style. Install with pip install ragas datasets openai.

1. Faithfulness + Answer Relevancy (no ground truth needed)

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy

data = {
    "question": [
        "How long does the thin-crust pizza take?",
        "How do I reset my password?"
    ],
    "answer": [
        "It is ready in 12–15 minutes.",
        "Passwords are very important for security."
    ],
    "contexts": [
        ["Our thin-crust pizza is ready in 12–15 minutes."],
        ["Password resets start from the login page via Forgot password."]
    ]
}

dataset = Dataset.from_dict(data)
result = evaluate(
    dataset,
    metrics=[faithfulness, answer_relevancy]
)
print(result)
print(result.to_pandas())

2. With ground truth (Context Recall + Answer Correctness)

from ragas.metrics import context_recall, answer_correctness

data = {
    "question": ["When was the first Super Bowl?"],
    "answer": ["The first Super Bowl was held on Jan 15, 1967"],
    "contexts": [[
        "The First AFL–NFL World Championship Game was played on January 15, 1967."
    ]],
    "ground_truth": ["The first Super Bowl was held on January 15, 1967"]
}

dataset = Dataset.from_dict(data)
result = evaluate(
    dataset,
    metrics=[context_recall, answer_correctness]
)
print(result)

3. Single-sample async style (newer API)

from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import Faithfulness

client = AsyncOpenAI()
llm = llm_factory("gpt-4o-mini", client=client)
scorer = Faithfulness(llm=llm)

result = await scorer.ascore(
    user_input="When was the first Super Bowl?",
    response="The first Super Bowl was held on Jan 15, 1967",
    retrieved_contexts=[
        "The First AFL–NFL World Championship Game was played on January 15, 1967."
    ]
)
print(f"Faithfulness: {result.value}")
Tip: Always look at the metrics together. High faithfulness + low context recall often means the model is carefully refusing to answer because the retriever failed.

Flashcards — test yourself

Quick mental models

  • Faithfulness → “No made-up facts”
  • Answer Relevancy → “Actually answered the question”
  • Context Precision → “Useful chunks, ranked high”
  • Context Recall → “Didn’t miss needed info”
  • Answer Correctness → “Close to the gold answer”
  • Recall@K → “Relevant docs in top-K?”
  • MRR → “How high is the first good hit?”
  • NDCG → “Best stuff ranked first?”

30-second summary

Ragas splits RAG evaluation into retrieval and generation. Use Faithfulness and Answer Relevancy when you have no gold answers. Add Context Recall and Answer Correctness when you do. Layer classic ranking metrics (Recall@K, MRR, NDCG) to debug the retriever. Never trust a single number — the combination tells you whether to fix the search or the prompt.
Ragas Metrics Blog · Funny analogies, real metrics, sample code. Always cross-check with the latest docs.ragas.io.

No comments:

Post a Comment