Debugging scenarioHard~15 minAI Evaluation & Observability / 17 of 18

Interview question

An AI feature's quality regressed in production. How would you diagnose it?

Reproduces failing cases, compares versioned traces, isolates the first changed component, mitigates impact, and adds regression coverage.

1Say itAnswer out loud first

2CompareCompare with Core answer

3RefineRefine with Stronger and Advanced

TL;DR

I first confirm the regression with representative failing examples and determine its scope, severity, start time, and affected slices. I compare candidate and baseline traces for model, prompt, context, retrieval, tools, validation, retries, and data versions to find the first stage where behavior diverges. If users face meaningful harm, I mitigate immediately through rollback, routing, fallback, or feature narrowing before completing the root-cause analysis.

Full answer below

Practice focus

Evidence-led incident diagnosis across model, prompt, data, tools, retrieval, graders, and traffic.

Answer first

Answer path

Core answer first, then checkpoints toward a stronger answer.

Start
Core answer
Start here. This is the direct spoken answer to practice first.
Overview
A quality incident needs the same disciplined isolation as a latency or correctness incident.
I first confirm the regression with representative failing examples and determine its scope, severity, start time, and affected slices. I compare candidate and baseline traces for model, prompt, context, retrieval, tools, validation, retries, and data versions to find the first stage where behavior diverges. If users face meaningful harm, I mitigate immediately through rollback, routing, fallback, or feature narrowing before completing the root-cause analysis.
Checkpoint
Stronger answer
Optional upgrade with examples, trade-offs, and sharper phrasing.
I verify that the evaluator and traffic distribution did not change, then replay redacted cases in a controlled environment when policy permits. Component evals distinguish a retrieval miss from generation misuse, a tool contract failure, a stale source, or a model change. Deployment markers, provider status, cost and latency shifts, feedback clusters, and recent configuration changes narrow the investigation.
Finish
Advanced answer
Optional deeper angle for production judgment and nuance.
The incident timeline preserves exact versions and evidence available at response time, because rerunning against today's sources can hide the original failure. I quantify blast radius and severity rather than extrapolating from the loudest report. The permanent fix is tested against the failing slice and the broader regression suite, then canaried with explicit stop conditions. Post-incident work improves observability, adds a case and release gate, and removes any manual diagnostic gap that delayed containment.

Review signals

Use these to catch weak phrasing, shallow reasoning, and risky claims.

Common mistakesCandidateCommon mistakesMistakes that weaken the answer or signal shallow understanding.Open details

Details

Changing the prompt before locating the failing stage.
Replaying against current rather than incident-time data.
Delaying rollback until root cause is proven.

Red flagsCandidateRed flagsSignals that an interviewer may treat as risky.Open details

Details

No trace can reproduce the execution path.
The grader change is ignored as a possible cause.
A one-case fix ships without broader regression testing.

Follow-up prompts

Quick interviewer prompts inside this drill.

Follow-up questionsCandidateFollow-up questionsLikely next questions after the main answer.Open details

Details

Treat these as the next interviewer turn. Answer one out loud, then save the gap in your practice notes.

What would make you roll back immediately?
How do you preserve incident-time evidence?
Which component eval would you run first?

Continue the section

Next in AI Evaluation & Observability: Design AI quality operations

Step 18 of 18

Design an evaluation and observability program for a production AI feature.

Practice route

Read core answer Review checklist Follow-up prompts Track progress

Finished this drill?

Practice freely. Sign in only when you want to save completion, confidence, notes, or bookmarks.

New to Aporeon? Create account

Drill details

Human editorial direction

Aporeon is shaped by Aleksandar Tomovski, a software developer with experience on both sides of technical interviews. Content is reviewed for accuracy, natural spoken delivery, useful depth, and honest trade-offs.

About the creator Editorial standards

Was this drill useful?