Build an offline AI evaluation harness with slice-based release gates

• Version the dataset, task contract, model route, prompt, tools, retrieval configuration, and graders.
• Run deterministic schema, citation, safety, and business-rule checks before probabilistic graders.
• Blind and randomize candidate/baseline order for pairwise model grading.
• Report global and named slice results with sample counts and uncertainty.
• Keep release thresholds tied to product risk and minimum effect size.
• Persist case-level evidence so failures can be reproduced and reviewed.

1. Load and validate an immutable dataset manifest with stable case IDs and declared slices.
2. Execute baseline and candidate configurations under fixed run budgets and capture full version metadata.
3. Apply deterministic checks, then grade remaining dimensions with calibrated graders and occasional human review.
4. Join results by case ID for paired comparisons and calculate slice metrics plus confidence intervals.
5. Evaluate explicit blocking and warning gates, including high-risk failure-rate ceilings.
6. Publish a report containing failing cases and configuration hashes, never just a single score.

def evaluate_release(dataset: Dataset, baseline: Config, candidate: Config) -> Decision:
    runs = {}
    for case in dataset.cases:
        runs[case.id] = PairedRun(
            baseline=execute(case, baseline),
            candidate=execute(case, candidate),
        )

    results = []
    for case in dataset.cases:
        pair = runs[case.id]
        deterministic = deterministic_checks(case, pair)
        judged = blinded_pairwise_grade(case, pair) if deterministic.can_grade else None
        results.append(case_result(case, pair, deterministic, judged))

    report = aggregate_paired(
        results,
        slices=dataset.declared_slices,
        confidence=0.95,
    )
    violations = [gate for gate in RELEASE_GATES if not gate.passes(report)]
    return Decision(
        allowed=not any(gate.blocking for gate in violations),
        violations=violations,
        report=report.with_reproducibility_manifest(dataset, baseline, candidate),
    )

Execution is O(cases × configurations), while model and grader calls dominate cost. Pairing reduces variance, but small slices still need explicit uncertainty and minimum sample handling.

• A grader prefers verbosity even though the task contract rewards concise supported answers.
• The candidate refuses more unsafe cases and appears worse on a naive answer-rate metric.
• A tiny high-risk slice produces an unstable percentage.
• The baseline and candidate use different retrieval corpora, invalidating an intended model-only comparison.
• Repeated near-duplicate cases overstate confidence.

• Improve the global mean while regressing a blocking risk slice and require a blocked decision.
• Swap candidate and baseline labels and expect a calibrated pairwise grader to remain stable.
• Remove version metadata and reject the run as non-reproducible.
• Inject schema-invalid output and ensure deterministic checks fail before model grading.
• Use a slice below its minimum sample size and report insufficient evidence rather than false precision.

• Optimizing one average score can hide severe regressions in important user groups or tasks.
• Using a model grader without calibration turns its preferences into an invisible product specification.
• Changing the dataset while comparing releases destroys the paired experiment.
• A pass/fail result without failing examples gives engineers nothing actionable.
• Treating evaluation as a one-time benchmark lets production behavior drift away from the release evidence.