Concept questionHard~12 minAI Evaluation & Observability / 6 of 18

Interview question

How do you avoid leakage and overfitting in an AI evaluation process?

Separates development and held-out cases, controls access, detects duplication, and validates improvements on fresh traffic.

1Say itAnswer out loud first

2CompareCompare with Core answer

3RefineRefine with Stronger and Advanced

TL;DR

I separate cases used during prompt development from a held-out set used for comparison, and I limit who or what can inspect the held-out answers. I check for duplicates and near-duplicates across training, tuning, and evaluation data where those sources are known. A change must improve representative slices, not merely the cases that motivated it.

Full answer below

Practice focus

Evaluation integrity, contamination, repeated tuning, and honest generalization claims.

Answer first

Answer path

Core answer first, then checkpoints toward a stronger answer.

Start
Core answer
Start here. This is the direct spoken answer to practice first.
Overview
An eval stops measuring generalization when the system is repeatedly optimized against the exact visible cases.
I separate cases used during prompt development from a held-out set used for comparison, and I limit who or what can inspect the held-out answers. I check for duplicates and near-duplicates across training, tuning, and evaluation data where those sources are known. A change must improve representative slices, not merely the cases that motivated it.
Checkpoint
Stronger answer
Optional upgrade with examples, trade-offs, and sharper phrasing.
A stable regression set is intentionally visible because it protects known failures, but passing it does not prove broader quality. I add fresh incident cases to a development pool first, then periodically rotate or expand held-out evaluation with independently labeled examples. Synthetic variants can test robustness, although superficial paraphrases do not create independent evidence.
Finish
Advanced answer
Optional deeper angle for production judgment and nuance.
I record how many tuning iterations have used each dataset and watch for a widening gap between visible regression results, held-out results, and sampled production evaluation. Access controls and audit trails reduce accidental leakage from benchmark answers into prompts or few-shot examples. When a provider model may already know a public benchmark, I avoid making product claims from that benchmark alone. Final rollout decisions combine held-out performance with guarded online evidence so the team cannot optimize indefinitely inside a closed test world.

Review signals

Use these to catch weak phrasing, shallow reasoning, and risky claims.

Common mistakesCandidateCommon mistakesMistakes that weaken the answer or signal shallow understanding.Open details

Details

Using one public dataset for development and final proof.
Calling paraphrases independent test cases.
Adding an incident to the test suite and claiming immediate generalization.

Red flagsCandidateRed flagsSignals that an interviewer may treat as risky.Open details

Details

Scores improve only on the visible regression set.
Held-out labels are embedded in prompt examples.
No one tracks dataset reuse or duplication.

Follow-up prompts

Quick interviewer prompts inside this drill.

Follow-up questionsCandidateFollow-up questionsLikely next questions after the main answer.Open details

Details

Treat these as the next interviewer turn. Answer one out loud, then save the gap in your practice notes.

What should remain visible to developers?
How often would you refresh held-out cases?
How would you detect benchmark contamination?

Continue the section

Next in AI Evaluation & Observability: Evaluate tools and outputs

Step 7 of 18

How would you evaluate structured output and tool use end to end?

Practice route

Read core answer Review checklist Follow-up prompts Track progress

Finished this drill?

Practice freely. Sign in only when you want to save completion, confidence, notes, or bookmarks.

New to Aporeon? Create account

Drill details

Human editorial direction

Aporeon is shaped by Aleksandar Tomovski, a software developer with experience on both sides of technical interviews. Content is reviewed for accuracy, natural spoken delivery, useful depth, and honest trade-offs.

About the creator Editorial standards

Was this drill useful?

Help improve the interview library.

Found this answer useful?

Say thanks with a standalone, one-time $5 contribution. No account required.

No subscription or perks.Choose another amount

Opening drill

Concept questionHard~12 minAI Evaluation & Observability / 6 of 18

Interview question

How do you avoid leakage and overfitting in an AI evaluation process?

Separates development and held-out cases, controls access, detects duplication, and validates improvements on fresh traffic.

1Say itAnswer out loud first

2CompareCompare with Core answer

3RefineRefine with Stronger and Advanced

TL;DR

Full answer below

Practice focus

Evaluation integrity, contamination, repeated tuning, and honest generalization claims.

Answer first

Answer path

Core answer first, then checkpoints toward a stronger answer.

Start
Core answer
Start here. This is the direct spoken answer to practice first.
Overview
An eval stops measuring generalization when the system is repeatedly optimized against the exact visible cases.
I separate cases used during prompt development from a held-out set used for comparison, and I limit who or what can inspect the held-out answers. I check for duplicates and near-duplicates across training, tuning, and evaluation data where those sources are known. A change must improve representative slices, not merely the cases that motivated it.
Checkpoint
Stronger answer
Optional upgrade with examples, trade-offs, and sharper phrasing.
A stable regression set is intentionally visible because it protects known failures, but passing it does not prove broader quality. I add fresh incident cases to a development pool first, then periodically rotate or expand held-out evaluation with independently labeled examples. Synthetic variants can test robustness, although superficial paraphrases do not create independent evidence.
Finish
Advanced answer
Optional deeper angle for production judgment and nuance.
I record how many tuning iterations have used each dataset and watch for a widening gap between visible regression results, held-out results, and sampled production evaluation. Access controls and audit trails reduce accidental leakage from benchmark answers into prompts or few-shot examples. When a provider model may already know a public benchmark, I avoid making product claims from that benchmark alone. Final rollout decisions combine held-out performance with guarded online evidence so the team cannot optimize indefinitely inside a closed test world.

Review signals

Use these to catch weak phrasing, shallow reasoning, and risky claims.

Common mistakesCandidateCommon mistakesMistakes that weaken the answer or signal shallow understanding.Open details

Details

Using one public dataset for development and final proof.
Calling paraphrases independent test cases.
Adding an incident to the test suite and claiming immediate generalization.

Red flagsCandidateRed flagsSignals that an interviewer may treat as risky.Open details

Details

Scores improve only on the visible regression set.
Held-out labels are embedded in prompt examples.
No one tracks dataset reuse or duplication.

Follow-up prompts

Quick interviewer prompts inside this drill.

Follow-up questionsCandidateFollow-up questionsLikely next questions after the main answer.Open details

Details

Treat these as the next interviewer turn. Answer one out loud, then save the gap in your practice notes.

What should remain visible to developers?
How often would you refresh held-out cases?
How would you detect benchmark contamination?

Continue the section

Next in AI Evaluation & Observability: Evaluate tools and outputs

Step 7 of 18

How would you evaluate structured output and tool use end to end?

How do you avoid leakage and overfitting in an AI evaluation process?

Practice focus

Answer path

Overview

Review signals

Follow-up prompts

Continue the section

How do you avoid leakage and overfitting in an AI evaluation process?

Practice focus

Answer path

Overview

Review signals

Follow-up prompts

Continue the section