Topic

AI Evaluation

Representative datasets, task metrics, graders, release comparisons, regressions, and feedback loops.

Related drills

Practice items tagged with AI Evaluation.

An AI feature became slower and more expensive. How would you diagnose it?

Diagnoses an AI regression by decomposing request stages, configuration changes, token growth, retries, routing, tools, retrieval, caching, and provider behavior.

Artificial IntelligenceLarge Language Models

Debugging scenarioHard~15 min

An AI feature's quality regressed in production. How would you diagnose it?

Reproduces failing cases, compares versioned traces, isolates the first changed component, mitigates impact, and adds regression coverage.

Artificial IntelligenceGenerative AILarge Language Models

Coding problemHard~40 min

Build an offline AI evaluation harness with slice-based release gates

Builds a reproducible evaluation harness that records immutable cases, versioned outputs, deterministic checks, calibrated graders, slices, and release decisions.

Artificial IntelligenceGenerative AILarge Language Models

Architecture scenarioHard~20 min

Design an evaluation and observability program for a production AI feature.

Connects product success, layered evals, calibrated graders, release gates, guarded rollout, traces, monitoring, feedback, and incident learning.

Artificial IntelligenceGenerative AILarge Language Models

Architecture scenarioHard~20 min

Design a production RAG answering boundary for internal knowledge.

Combines governed ingestion, authorized hybrid retrieval, reranking, grounded generation, citations, evaluation, observability, and fallback.

Artificial IntelligenceGenerative AILarge Language ModelsVector Databases

Coding problemHard~35 min

Diagnose an agent loop that repeats tools and exhausts its budget

Uses workflow traces to distinguish planning failure, stale observations, retry ambiguity, and missing terminal conditions in a looping agent.

Artificial IntelligenceGenerative AILarge Language Models