Skip to content

Tracks Drills Topics Technologies Tech stacks Skill areas

Tracks Drills Topics

Loading

Aporeon

Technical interview practice, from one developer to another.

About Editorial standards Support Privacy Terms Refund policy

AI Evaluation & Observability

Datasets, graders, traces, quality metrics, regressions, production feedback, and release evidence.

18 drillsPractice one at a timeOrdered pathConcept questionArchitecture scenarioComparison questionDebugging scenario

Recommended start

What AI evals test

Defines AI evaluation as repeatable measurement of application behavior while preserving deterministic tests for code and contracts.

Easy~7 minStep 1 of 18

Practice this drill Choose another

Search

18 of 18 shown

1What AI evals testRecommended startDefines AI evaluation as repeatable measurement of application behavior while preserving deterministic tests for code and contracts.Concept questionEasy~7 minArtificial IntelligenceGenerative AILarge Language ModelsPractice 2Define AI task successTurns a product outcome into observable quality, safety, and operational criteria before selecting convenient scores.Architecture scenarioMedium~10 minArtificial IntelligenceGenerative AILarge Language ModelsPractice 3Build an AI eval datasetCombines production-shaped, expert-authored, edge, and adversarial cases with versioned labels and privacy controls.Architecture scenarioMedium~12 minArtificial IntelligenceGenerative AILarge Language ModelsPractice 4Choose an AI graderMatches exact checks, model judgment, and human expertise to the behavior being measured instead of treating graders as interchangeable.Comparison questionMedium~10 minArtificial IntelligenceGenerative AILarge Language ModelsPractice 5Calibrate an LLM judgeBuilds a focused scoring guide, representative labeled examples, bias checks, and ongoing agreement monitoring for model-based evaluation.Architecture scenarioHard~15 minArtificial IntelligenceGenerative AILarge Language ModelsPractice 6Avoid eval overfittingSeparates development and held-out cases, controls access, detects duplication, and validates improvements on fresh traffic.Concept questionHard~12 minArtificial IntelligenceGenerative AILarge Language ModelsPractice 7Evaluate tools and outputsMeasures schema validity, tool selection, argument correctness, policy, execution outcome, repair, and final user result.Architecture scenarioHard~15 minArtificial IntelligenceGenerative AILarge Language ModelsPractice 8Evaluate grounding and abstentionJudges claims against allowed evidence, separates correctness from support, and tests whether the system answers or abstains appropriately.Architecture scenarioHard~15 minArtificial IntelligenceGenerative AILarge Language ModelsPractice 9Pairwise vs absolute scoringUses pairwise judgments for relative change and absolute criteria for release obligations while controlling order and verbosity bias.Comparison questionMedium~10 minArtificial IntelligenceGenerative AILarge Language ModelsPractice 10Evaluate AI behavior by sliceBreaks results down by meaningful user, task, risk, language, input, and system dimensions instead of trusting one average.Concept questionMedium~10 minArtificial IntelligenceGenerative AILarge Language ModelsPractice 11Set AI release gatesCombines deterministic invariants, severity-aware quality thresholds, slice protection, baseline comparison, and guarded rollout.Architecture scenarioHard~15 minArtificial IntelligenceGenerative AILarge Language ModelsPractice 12Roll out AI changes safelySelects a production experiment based on user exposure, side effects, measurement needs, cost, and rollback risk.Comparison questionHard~15 minArtificial IntelligenceGenerative AILarge Language ModelsPractice 13Trace an AI requestConnects application, model, retrieval, tools, validation, retries, and outcomes while controlling sensitive telemetry.Architecture scenarioMedium~12 minArtificial IntelligenceGenerative AILarge Language ModelsPractice 14Monitor production AI metricsBalances task quality, safety, reliability, latency, tokens, cost, refusals, repairs, and fallbacks by route and slice.Concept questionMedium~10 minArtificial IntelligenceGenerative AILarge Language ModelsPractice 15Use AI product feedbackCombines low-friction ratings, structured reasons, behavioral outcomes, review queues, and privacy-aware eval case creation.Architecture scenarioMedium~12 minArtificial IntelligenceGenerative AILarge Language ModelsPractice 16Detect AI behavior driftSeparates traffic, data, dependency, model, prompt, and grader change while maintaining stable anchors and fresh evaluation.Architecture scenarioHard~15 minArtificial IntelligenceGenerative AILarge Language ModelsPractice 17Diagnose an AI regressionReproduces failing cases, compares versioned traces, isolates the first changed component, mitigates impact, and adds regression coverage.Debugging scenarioHard~15 minArtificial IntelligenceGenerative AILarge Language ModelsPractice 18Design AI quality operationsConnects product success, layered evals, calibrated graders, release gates, guarded rollout, traces, monitoring, feedback, and incident learning.Architecture scenarioHard~20 minArtificial IntelligenceGenerative AILarge Language ModelsPractice

Practice order

18 drillsOne at a time

1What AI evals testUp next / Easy / ~7 min
2Define AI task successMedium / ~10 min
3Build an AI eval datasetMedium / ~12 min
4Choose an AI graderMedium / ~10 min
5Calibrate an LLM judgeHard / ~15 min
6Avoid eval overfittingHard / ~12 min
7Evaluate tools and outputsHard / ~15 min
8Evaluate grounding and abstentionHard / ~15 min

+10 more drills in this section