Architecture scenarioHard~12 minAI Fundamentals / 18 of 20

Interview question

How would you roll out a model or model-version change safely?

Treats model changes as controlled releases using versioned configuration, fixed evaluations, shadow or canary traffic, monitoring, and rollback.

1Say itAnswer out loud first

2CompareCompare with Core answer

3RefineRefine with Stronger and Advanced

TL;DR

I version the current model, prompt, tools, retrieval settings, and decoding configuration, then run the candidate against the same representative evaluation set. I compare task quality, important failure slices, invalid outputs, refusals, latency, and cost. If the candidate clears the gate, I use shadow traffic where privacy and cost allow, or a small canary, and monitor both technical and task metrics. The previous configuration remains available for fast rollback.

Full answer below

Practice focus

Release discipline for dependencies whose semantic behavior can change even when the API contract remains compatible.

Answer first

Answer path

Core answer first, then checkpoints toward a stronger answer.

Start
Core answer
Start here. This is the direct spoken answer to practice first.
Overview
A model upgrade is a behavior change, so a successful API call is not enough evidence to promote it.
I version the current model, prompt, tools, retrieval settings, and decoding configuration, then run the candidate against the same representative evaluation set. I compare task quality, important failure slices, invalid outputs, refusals, latency, and cost. If the candidate clears the gate, I use shadow traffic where privacy and cost allow, or a small canary, and monitor both technical and task metrics. The previous configuration remains available for fast rollback.
Checkpoint
Stronger answer
Optional upgrade with examples, trade-offs, and sharper phrasing.
The evaluation includes recent production examples that are safe to retain, adversarial cases, and known past failures. I avoid changing the model and several prompts at once unless the release is intentionally evaluated as one bundle. During a canary, users are assigned consistently when conversation behavior matters. I also check context limits, tokenizer changes, schema and tool support, provider quotas, and regional availability.
Finish
Advanced answer
Optional deeper angle for production judgment and nuance.
Promotion criteria and stop conditions are agreed before traffic moves. I compare distributions and slices rather than one average score, investigate disagreement between automated graders and human review, and account for provider-side aliases that may change underneath a friendly model name. The registry records exact identifiers and release evidence. If rollback affects cached results, embeddings, or persisted workflow state, the plan covers compatibility instead of assuming the model is a stateless swap.

Review signals

Use these to catch weak phrasing, shallow reasoning, and risky claims.

Common mistakesCandidateCommon mistakesMistakes that weaken the answer or signal shallow understanding.Open details

Details

Replacing a model globally after a few manual prompts.
Using a mutable provider alias without recording the resolved version.
Changing model, prompt, and retrieval together with no comparative baseline.
Monitoring only HTTP errors after release.

Red flagsCandidateRed flagsSignals that an interviewer may treat as risky.Open details

Details

There is no previous configuration to restore.
Canary success ignores semantic quality and cost.
Persisted state becomes unreadable after rollback.

Follow-up prompts

Quick interviewer prompts inside this drill.

Follow-up questionsCandidateFollow-up questionsLikely next questions after the main answer.Open details

Details

Treat these as the next interviewer turn. Answer one out loud, then save the gap in your practice notes.

What belongs in the release bundle?
Which metrics should stop a canary?
When is shadow traffic useful?
How would you handle provider model retirement?

Continue the section

Next in AI Fundamentals: Diagnose AI latency and cost

Step 19 of 20

An AI feature became slower and more expensive. How would you diagnose it?

Practice route

Read core answer Review checklist Follow-up prompts Track progress

Finished this drill?

Practice freely. Sign in only when you want to save completion, confidence, notes, or bookmarks.

New to Aporeon? Create account

Drill details

Human editorial direction

Aporeon is shaped by Aleksandar Tomovski, a software developer with experience on both sides of technical interviews. Content is reviewed for accuracy, natural spoken delivery, useful depth, and honest trade-offs.

About the creator Editorial standards

Was this drill useful?