Concept questionEasy~7 minAI Evaluation & Observability / 1 of 18

Interview question

What is an AI evaluation, and why are normal unit tests not enough?

Defines AI evaluation as repeatable measurement of application behavior while preserving deterministic tests for code and contracts.

1Say itAnswer out loud first

2CompareCompare with Core answer

3RefineRefine with Stronger and Advanced

TL;DR

An AI evaluation is a repeatable test of whether an AI feature behaves well on representative inputs and defined success criteria. Normal unit tests still verify code paths, schemas, permissions, and deterministic rules, but they cannot fully judge whether an open-ended response is useful, supported, or appropriately cautious. I use both: conventional tests protect software contracts, while evals measure the variable behavior of the model and the application around it.

Full answer below

Practice focus

Whether the candidate separates deterministic correctness from probabilistic output quality.

Answer first

Answer path

Core answer first, then checkpoints toward a stronger answer.

Start
Core answer
Start here. This is the direct spoken answer to practice first.
Overview
AI features still need normal tests, but generated language introduces quality questions that exact assertions cannot answer alone.
An AI evaluation is a repeatable test of whether an AI feature behaves well on representative inputs and defined success criteria. Normal unit tests still verify code paths, schemas, permissions, and deterministic rules, but they cannot fully judge whether an open-ended response is useful, supported, or appropriately cautious. I use both: conventional tests protect software contracts, while evals measure the variable behavior of the model and the application around it.
Checkpoint
Stronger answer
Optional upgrade with examples, trade-offs, and sharper phrasing.
For a support assistant, unit tests can prove that an unauthorized tool is never called and that malformed JSON is rejected. An eval set can then measure whether answers resolve the user's issue, stay within policy, cite evidence when required, and abstain when the information is missing. That split matters because a perfectly valid response object can still contain a poor answer, while a fluent answer can still violate an application rule.
Finish
Advanced answer
Optional deeper angle for production judgment and nuance.
I define evals at every nondeterministic boundary rather than assigning one vague quality score to the whole system. Component checks can isolate classification, retrieval, tool selection, structured output, and final response quality, while end-to-end cases verify the user outcome. Scores are useful only when their dataset, scoring guide, grader, and application version are reproducible. Production failures and human review continuously add cases, so the suite evolves without replacing incident monitoring or deterministic safety controls.

Review signals

Use these to catch weak phrasing, shallow reasoning, and risky claims.

Common mistakesCandidateCommon mistakesMistakes that weaken the answer or signal shallow understanding.Open details

Details

Replacing unit tests with model-based grading.
Calling a handful of prompt experiments an evaluation suite.
Measuring fluency while ignoring task success.

Red flagsCandidateRed flagsSignals that an interviewer may treat as risky.Open details

Details

A valid JSON response is treated as a correct answer.
The team cannot state what the eval score represents.
Only the final response is tested in a multi-stage system.

Follow-up prompts

Quick interviewer prompts inside this drill.

Follow-up questionsCandidateFollow-up questionsLikely next questions after the main answer.Open details

Details

Treat these as the next interviewer turn. Answer one out loud, then save the gap in your practice notes.

Where would you place component evals?
Which behaviors should remain deterministic?
How would production incidents change the suite?

Continue the section

Next in AI Evaluation & Observability: Define AI task success

Step 2 of 18

How would you define success for an AI feature before choosing metrics?

Practice route

Read core answer Review checklist Follow-up prompts Track progress

Finished this drill?

Practice freely. Sign in only when you want to save completion, confidence, notes, or bookmarks.

New to Aporeon? Create account

Drill details

Human editorial direction

Aporeon is shaped by Aleksandar Tomovski, a software developer with experience on both sides of technical interviews. Content is reviewed for accuracy, natural spoken delivery, useful depth, and honest trade-offs.

About the creator Editorial standards

Was this drill useful?

Help improve the interview library.

Found this answer useful?

Say thanks with a standalone, one-time $5 contribution. No account required.

No subscription or perks.Choose another amount

Opening drill

Concept questionEasy~7 minAI Evaluation & Observability / 1 of 18

Interview question

What is an AI evaluation, and why are normal unit tests not enough?

Defines AI evaluation as repeatable measurement of application behavior while preserving deterministic tests for code and contracts.

1Say itAnswer out loud first

2CompareCompare with Core answer

3RefineRefine with Stronger and Advanced

TL;DR

Full answer below

Practice focus

Whether the candidate separates deterministic correctness from probabilistic output quality.

Answer first

Answer path

Core answer first, then checkpoints toward a stronger answer.

Start
Core answer
Start here. This is the direct spoken answer to practice first.
Overview
AI features still need normal tests, but generated language introduces quality questions that exact assertions cannot answer alone.
An AI evaluation is a repeatable test of whether an AI feature behaves well on representative inputs and defined success criteria. Normal unit tests still verify code paths, schemas, permissions, and deterministic rules, but they cannot fully judge whether an open-ended response is useful, supported, or appropriately cautious. I use both: conventional tests protect software contracts, while evals measure the variable behavior of the model and the application around it.
Checkpoint
Stronger answer
Optional upgrade with examples, trade-offs, and sharper phrasing.
For a support assistant, unit tests can prove that an unauthorized tool is never called and that malformed JSON is rejected. An eval set can then measure whether answers resolve the user's issue, stay within policy, cite evidence when required, and abstain when the information is missing. That split matters because a perfectly valid response object can still contain a poor answer, while a fluent answer can still violate an application rule.
Finish
Advanced answer
Optional deeper angle for production judgment and nuance.
I define evals at every nondeterministic boundary rather than assigning one vague quality score to the whole system. Component checks can isolate classification, retrieval, tool selection, structured output, and final response quality, while end-to-end cases verify the user outcome. Scores are useful only when their dataset, scoring guide, grader, and application version are reproducible. Production failures and human review continuously add cases, so the suite evolves without replacing incident monitoring or deterministic safety controls.

Review signals

Use these to catch weak phrasing, shallow reasoning, and risky claims.

Common mistakesCandidateCommon mistakesMistakes that weaken the answer or signal shallow understanding.Open details

Details

Replacing unit tests with model-based grading.
Calling a handful of prompt experiments an evaluation suite.
Measuring fluency while ignoring task success.

Red flagsCandidateRed flagsSignals that an interviewer may treat as risky.Open details

Details

A valid JSON response is treated as a correct answer.
The team cannot state what the eval score represents.
Only the final response is tested in a multi-stage system.

Follow-up prompts

Quick interviewer prompts inside this drill.

Follow-up questionsCandidateFollow-up questionsLikely next questions after the main answer.Open details

Details

Treat these as the next interviewer turn. Answer one out loud, then save the gap in your practice notes.

Where would you place component evals?
Which behaviors should remain deterministic?
How would production incidents change the suite?

Continue the section

Next in AI Evaluation & Observability: Define AI task success

Step 2 of 18

How would you define success for an AI feature before choosing metrics?

What is an AI evaluation, and why are normal unit tests not enough?

Practice focus

Answer path

Overview

Review signals

Follow-up prompts

Continue the section

What is an AI evaluation, and why are normal unit tests not enough?

Practice focus

Answer path

Overview

Review signals

Follow-up prompts

Continue the section