Diagnose a RAG quality regression from retrieval and generation evidence

• Do not blame the model before separating retrieval, context, and generation behavior.
• Use production-safe identifiers and approved replay data rather than copying customer prompts.
• Compare the regressed slice against a stable control slice.
• Choose an immediate mitigation that is reversible and does not corrupt newer index data.
• Define a verification gate for the repaired pipeline.

1. Confirm the measurement is real by checking evaluator version, sample composition, confidence intervals, and traffic mix.
2. Replay labeled queries through each pipeline stage and compare candidate recall before and after fusion and reranking.
3. Run generation with oracle context to test whether the answer stage still performs when retrieval is correct.
4. Correlate failures with ingestion time, chunker, embedding, namespace, and document freshness.
5. Freeze or roll back the alias to the last known-good index while preserving the failed index for analysis.
6. Rebuild a small affected corpus, pass slice-level retrieval and groundedness gates, then promote through a canary.

async def diagnose(cases: list[EvalCase], old: Pipeline, new: Pipeline) -> Report:
    report = Report()
    for case in cases:
        before = await old.trace(case.question, actor=case.actor)
        after = await new.trace(case.question, actor=case.actor)
        report.add(
            slice=case.slice,
            retrieval_before=recall_at_k(before.candidates, case.relevant_chunks, 10),
            retrieval_after=recall_at_k(after.candidates, case.relevant_chunks, 10),
            context_before=context_recall(before.context, case.relevant_chunks),
            context_after=context_recall(after.context, case.relevant_chunks),
            answer_after=groundedness(after.answer, after.context),
            oracle_after=groundedness(
                await new.generate(case.question, case.oracle_context),
                case.oracle_context,
            ),
            versions=after.versions,
        )
    return report.compare_by_slice_and_version()

The diagnostic replay is O(cases × compared pipeline stages) and should run on a bounded labeled set. Production mitigation must not depend on completing a full-corpus experiment first.

• The evaluator changed at the same time as the pipeline and creates an apparent regression.
• Only documents updated after a timestamp are missing because incremental ingestion stopped.
• Retrieval finds relevant chunks, but context assembly drops them during deduplication.
• The new embedding index cannot be queried correctly by the old embedding model after a partial rollback.
• Latency improved because a broken namespace returns fewer candidates.

• Provide strong oracle-context results and expect the diagnosis to prioritize retrieval over generation.
• Change the evaluator version only and verify the plan checks measurement comparability.
• Create a regression isolated to one tenant or content source and require slice-level analysis.
• Make the old and new embedding spaces incompatible and reject an unsafe component-only rollback.
• Require a reversible alias or routing mitigation before destructive reindexing.

• Changing prompts and models immediately adds variables before locating the broken stage.
• Looking only at aggregate answer quality hides the recently updated policy slice.
• Rebuilding the production index in place destroys evidence and removes rollback options.
• Using oracle context as the final solution ignores the retrieval path users actually depend on.
• Declaring success on a handful of anecdotes without a versioned evaluation gate repeats the incident.