Architecture scenarioHard~15 minRAG & Vector Search / 13 of 18

Interview question

How would you evaluate retrieval separately from generation in a RAG system?

Builds a query-and-relevance dataset and separates retrieval recall, ranking quality, context quality, and generated-answer quality.

1Say itAnswer out loud first

2CompareCompare with Core answer

3RefineRefine with Stronger and Advanced

TL;DR

I create representative queries with relevant document or chunk judgments, including valid no-answer cases. Retrieval metrics such as recall at k, precision at k, mean reciprocal rank, or normalized discounted cumulative gain measure whether useful evidence was found and ordered. Generation is then evaluated on answer correctness, support, completeness, and citation behavior using a fixed retrieved context.

Full answer below

Practice focus

Evaluation decomposition, labeled evidence, slices, regressions, and metric limits.

Answer first

Answer path

Core answer first, then checkpoints toward a stronger answer.

Start
Core answer
Start here. This is the direct spoken answer to practice first.
Overview
End-to-end answer scores alone cannot tell whether retrieval or generation caused a failure.
I create representative queries with relevant document or chunk judgments, including valid no-answer cases. Retrieval metrics such as recall at k, precision at k, mean reciprocal rank, or normalized discounted cumulative gain measure whether useful evidence was found and ordered. Generation is then evaluated on answer correctness, support, completeness, and citation behavior using a fixed retrieved context.
Checkpoint
Stronger answer
Optional upgrade with examples, trade-offs, and sharper phrasing.
The dataset includes exact identifiers, paraphrases, multi-part questions, stale documents, permission boundaries, rare topics, and ambiguous requests. I inspect results by slice because an aggregate score can hide a severe regression for one language or document type. Human judgments remain the reference for difficult relevance and support decisions, while automated graders are calibrated and monitored against them.
Finish
Advanced answer
Optional deeper angle for production judgment and nuance.
I version the corpus snapshot, queries, judgments, chunker, embedding model, index configuration, and retrieval policy so comparisons are meaningful. Retrieval metrics have limits: recall may improve by returning many noisy chunks, while reciprocal rank may miss multi-source coverage. I add task-specific measures such as evidence coverage, duplicate rate, authorized recall, abstention correctness, latency, and cost. Release gates use a small stable regression set plus broader exploratory evaluation, followed by guarded online measurement.

Review signals

Use these to catch weak phrasing, shallow reasoning, and risky claims.

Common mistakesCandidateCommon mistakesMistakes that weaken the answer or signal shallow understanding.Open details

Details

Scoring only the final generated answer.
Building a dataset with no no-answer or permission cases.
Comparing retrievers against different corpus versions.

Red flagsCandidateRed flagsSignals that an interviewer may treat as risky.Open details

Details

A retrieval regression is fixed by changing the answer prompt.
Automated grader scores have never been checked by humans.
One aggregate metric controls release.

Follow-up prompts

Quick interviewer prompts inside this drill.

Follow-up questionsCandidateFollow-up questionsLikely next questions after the main answer.Open details

Details

Treat these as the next interviewer turn. Answer one out loud, then save the gap in your practice notes.

Which metric fits a single-answer lookup?
How would you label multi-source questions?
What belongs in a retrieval regression set?

Continue the section

Next in RAG & Vector Search: Keep a RAG index fresh

Step 14 of 18

How would you keep a RAG index fresh through updates and deletes?

Practice route

Read core answer Review checklist Follow-up prompts Track progress

Finished this drill?

Practice freely. Sign in only when you want to save completion, confidence, notes, or bookmarks.

New to Aporeon? Create account

Drill details

Human editorial direction

Aporeon is shaped by Aleksandar Tomovski, a software developer with experience on both sides of technical interviews. Content is reviewed for accuracy, natural spoken delivery, useful depth, and honest trade-offs.

About the creator Editorial standards

Was this drill useful?