A RAG system missed the right document. How would you diagnose it?

Traces a missed answer through source ingestion, parsing, chunking, filtering, query construction, retrieval, fusion, and ranking.

1Say itAnswer out loud first

2CompareCompare with Core answer

3RefineRefine with Stronger and Advanced

TL;DR

I start with one failing query and the expected source, then check whether the correct source version was discovered, parsed, chunked, embedded, and present in the active index. Next I inspect mandatory filters, the original and rewritten queries, keyword and vector candidate lists, fusion, reranking, and the final cutoff. I change the first failing stage instead of tuning the answer prompt.

Full answer below

Practice focus

Evidence-led diagnosis across the full retrieval pipeline.

Answer first

Answer path

Core answer first, then checkpoints toward a stronger answer.

Start
Core answer
Start here. This is the direct spoken answer to practice first.
Overview
A missing result can originate anywhere from source discovery to final top-k selection.
I start with one failing query and the expected source, then check whether the correct source version was discovered, parsed, chunked, embedded, and present in the active index. Next I inspect mandatory filters, the original and rewritten queries, keyword and vector candidate lists, fusion, reranking, and the final cutoff. I change the first failing stage instead of tuning the answer prompt.
Checkpoint
Stronger answer
Optional upgrade with examples, trade-offs, and sharper phrasing.
If the chunk exists but never enters the candidate set, I compare lexical terms, embedding similarity, metadata, and query preprocessing. If it appears early and disappears later, I inspect score fusion, duplicate handling, reranker truncation, thresholds, and top-k. I reproduce against a versioned corpus and configuration so a live index change does not move the evidence during diagnosis.

Review signals

Use these to catch weak phrasing, shallow reasoning, and risky claims.

Common mistakesCandidateCommon mistakesMistakes that weaken the answer or signal shallow understanding.Open details

Details

Editing the generation prompt before checking retrieval.
Looking only at the final five chunks.
Testing against a different corpus version.

A RAG system missed the right document. How would you diagnose it?

Practice focus

Answer path

Overview

Review signals

Follow-up prompts

Continue practice

Next in path

Tech stacks

Skill areas

Human editorial direction

Was this drill useful?

Found this answer useful?