Comparison questionHard~15 minAI Evaluation & Observability / 12 of 18

Interview question

How would you use shadow, canary, and A/B testing for an AI change?

Selects a production experiment based on user exposure, side effects, measurement needs, cost, and rollback risk.

1Say itAnswer out loud first

2CompareCompare with Core answer

3RefineRefine with Stronger and Advanced

TL;DR

Shadow testing sends a copy of real requests to the candidate without using its response, so it is good for compatibility, latency, and offline quality comparison with low user risk. A canary exposes a small controlled population and is best for detecting production failures before broad rollout. An A/B test randomizes eligible users between variants when the goal is to measure a user or business outcome rather than only technical correctness.

Full answer below

Practice focus

Online experiment design, safety, attribution, side effects, and operational rollout.

Answer first

Answer path

Core answer first, then checkpoints toward a stronger answer.

Start
Core answer
Start here. This is the direct spoken answer to practice first.
Overview
Online evaluation provides real behavior, but each rollout method exposes users and systems differently.
Shadow testing sends a copy of real requests to the candidate without using its response, so it is good for compatibility, latency, and offline quality comparison with low user risk. A canary exposes a small controlled population and is best for detecting production failures before broad rollout. An A/B test randomizes eligible users between variants when the goal is to measure a user or business outcome rather than only technical correctness.
Checkpoint
Stronger answer
Optional upgrade with examples, trade-offs, and sharper phrasing.
Shadow paths must suppress state-changing tools, protect data, and account for doubled provider cost. Canaries need explicit stop thresholds for quality, errors, latency, spend, and safety plus fast rollback. A/B tests require stable assignment, a defined primary outcome, guardrail metrics, enough duration, and protection against cross-variant conversation state.
Finish
Advanced answer
Optional deeper angle for production judgment and nuance.
I choose the least risky method that can answer the question and often use them in sequence: shadow, internal canary, external canary, then broader experiment. Production graders are sampled and calibrated rather than trusted as the sole outcome, while delayed user outcomes are joined back to the correct model, prompt, tool, and data versions. Novel failures enter the offline suite before the next release. High-risk actions may never be eligible for uncontrolled experimentation even when the language response is randomized.

Review signals

Use these to catch weak phrasing, shallow reasoning, and risky claims.

Common mistakesCandidateCommon mistakesMistakes that weaken the answer or signal shallow understanding.Open details

Details

Allowing a shadow path to execute real side effects.
Using clicks as the only A/B outcome.
Running a canary without automatic stop conditions.

Red flagsCandidateRed flagsSignals that an interviewer may treat as risky.Open details

Details

Variant assignment changes mid-conversation.
The candidate doubles cost without a budget limit.
No trace identifies which variant produced an outcome.

Follow-up prompts

Quick interviewer prompts inside this drill.

Follow-up questionsCandidateFollow-up questionsLikely next questions after the main answer.Open details

Details

Treat these as the next interviewer turn. Answer one out loud, then save the gap in your practice notes.

What cannot safely run in shadow mode?
Which metrics would stop a canary?
How would you handle delayed outcomes?

Continue the section

Next in AI Evaluation & Observability: Trace an AI request

Step 13 of 18

What would you trace in a production AI request?

Practice route

Read core answer Review checklist Follow-up prompts Track progress

Finished this drill?

Practice freely. Sign in only when you want to save completion, confidence, notes, or bookmarks.

New to Aporeon? Create account

Drill details

Human editorial direction

Aporeon is shaped by Aleksandar Tomovski, a software developer with experience on both sides of technical interviews. Content is reviewed for accuracy, natural spoken delivery, useful depth, and honest trade-offs.

About the creator Editorial standards

Was this drill useful?

Help improve the interview library.

Found this answer useful?

Say thanks with a standalone, one-time $5 contribution. No account required.

No subscription or perks.Choose another amount

Opening drill

Comparison questionHard~15 minAI Evaluation & Observability / 12 of 18

Interview question

How would you use shadow, canary, and A/B testing for an AI change?

Selects a production experiment based on user exposure, side effects, measurement needs, cost, and rollback risk.

1Say itAnswer out loud first

2CompareCompare with Core answer

3RefineRefine with Stronger and Advanced

TL;DR

Full answer below

Practice focus

Online experiment design, safety, attribution, side effects, and operational rollout.

Answer first

Answer path

Core answer first, then checkpoints toward a stronger answer.

Start
Core answer
Start here. This is the direct spoken answer to practice first.
Overview
Online evaluation provides real behavior, but each rollout method exposes users and systems differently.
Shadow testing sends a copy of real requests to the candidate without using its response, so it is good for compatibility, latency, and offline quality comparison with low user risk. A canary exposes a small controlled population and is best for detecting production failures before broad rollout. An A/B test randomizes eligible users between variants when the goal is to measure a user or business outcome rather than only technical correctness.
Checkpoint
Stronger answer
Optional upgrade with examples, trade-offs, and sharper phrasing.
Shadow paths must suppress state-changing tools, protect data, and account for doubled provider cost. Canaries need explicit stop thresholds for quality, errors, latency, spend, and safety plus fast rollback. A/B tests require stable assignment, a defined primary outcome, guardrail metrics, enough duration, and protection against cross-variant conversation state.
Finish
Advanced answer
Optional deeper angle for production judgment and nuance.
I choose the least risky method that can answer the question and often use them in sequence: shadow, internal canary, external canary, then broader experiment. Production graders are sampled and calibrated rather than trusted as the sole outcome, while delayed user outcomes are joined back to the correct model, prompt, tool, and data versions. Novel failures enter the offline suite before the next release. High-risk actions may never be eligible for uncontrolled experimentation even when the language response is randomized.

Review signals

Use these to catch weak phrasing, shallow reasoning, and risky claims.

Common mistakesCandidateCommon mistakesMistakes that weaken the answer or signal shallow understanding.Open details

Details

Allowing a shadow path to execute real side effects.
Using clicks as the only A/B outcome.
Running a canary without automatic stop conditions.

Red flagsCandidateRed flagsSignals that an interviewer may treat as risky.Open details

Details

Variant assignment changes mid-conversation.
The candidate doubles cost without a budget limit.
No trace identifies which variant produced an outcome.

Follow-up prompts

Quick interviewer prompts inside this drill.

Follow-up questionsCandidateFollow-up questionsLikely next questions after the main answer.Open details

Details

Treat these as the next interviewer turn. Answer one out loud, then save the gap in your practice notes.

What cannot safely run in shadow mode?
Which metrics would stop a canary?
How would you handle delayed outcomes?

Continue the section

Next in AI Evaluation & Observability: Trace an AI request

Step 13 of 18

What would you trace in a production AI request?

How would you use shadow, canary, and A/B testing for an AI change?

Practice focus

Answer path

Overview

Review signals

Follow-up prompts

Continue the section

How would you use shadow, canary, and A/B testing for an AI change?

Practice focus

Answer path

Overview

Review signals

Follow-up prompts

Continue the section