Build hybrid RAG retrieval with access control and verified citations

• Apply tenant and document authorization in both lexical and vector retrieval, not after top-k selection.
• Fuse rankings using stable document/chunk identities and remove duplicate chunks.
• Keep the final context within a configured token budget and preserve provenance metadata.
• Ask for explicit chunk identifiers in the generated answer.
• Verify every citation against the supplied context and abstain when material claims lack support.
• Measure retrieval recall and answer groundedness separately.

1. Normalize the query and create one immutable authorization scope from trusted caller context.
2. Run lexical and vector searches with the same scope and a larger candidate budget than the final context.
3. Fuse ranks, deduplicate by canonical chunk identity, and optionally rerank the authorized candidate set.
4. Select diverse chunks within the context budget and serialize text separately from trusted provenance.
5. Generate an answer constrained to supplied chunk IDs, then verify citation membership and textual support.
6. Return the answer only when verification passes; otherwise abstain or route to review with diagnostic reason codes.

async def answer_policy_question(question: str, actor: Principal) -> RagResult:
    scope = await permissions.authorized_document_scope(actor)
    lexical, semantic = await gather(
        search.lexical(question, tenant=actor.tenant_id, document_ids=scope, limit=30),
        search.vector(question, tenant=actor.tenant_id, document_ids=scope, limit=30),
    )

    candidates = reciprocal_rank_fusion(lexical, semantic)
    candidates = deduplicate_chunks(candidates)
    ranked = await reranker.rank(question, candidates[:40])
    context = select_within_budget(ranked, token_budget=6_000, diversify_by_document=True)

    if not context:
        return RagResult.abstain("no_authorized_evidence")

    generated = await model.answer(
        question=question,
        chunks=[{"id": c.id, "text": c.text} for c in context],
        require_citations=True,
    )
    verification = citation_verifier.check(generated, context)
    if not verification.material_claims_supported:
        return RagResult.abstain("unsupported_citation", diagnostics=verification.safe_codes)
    return RagResult.answer(generated.text, verification.citations)

Retrieval cost depends on the search indexes; fusion is O(k log k) for bounded candidate sets, and reranking/model calls dominate latency. Authorization scope size must be represented efficiently rather than copied into an unbounded filter.

• The vector result and lexical result refer to the same canonical chunk under different index records.
• The only highly similar document belongs to another tenant.
• A cited chunk exists in context but does not support the sentence attached to it.
• Several individually useful chunks exceed the context budget and require deterministic selection.
• A recently revoked document remains in one index replica.

• Seed a cross-tenant near-duplicate and assert it never reaches fusion, reranking, prompts, or logs.
• Return overlapping lexical and vector candidates and verify stable deduplication.
• Generate a citation to a chunk that was retrieved but excluded from final context and reject it.
• Remove all supporting chunks and verify an abstention rather than an uncited answer.
• Evaluate retrieval recall independently from generation using a labeled question set.

• Filtering unauthorized results after top-k retrieval can both leak data and crowd out valid candidates.
• Treating a citation-shaped token as evidence allows fabricated or irrelevant citations.
• Optimizing only final answer scores makes retrieval regressions hard to locate.
• Stuffing every candidate into the prompt wastes context and increases distraction.
• Using mutable display titles as provenance keys breaks verification after content changes.