Concept questionMedium~10 minPrompting & Structured Output / 9 of 20

Interview question

Why can schema-valid model output still be wrong?

Separates structural validity from factual accuracy, evidence, authorization, and domain invariants.

1Say itAnswer out loud first

2CompareCompare with Core answer

3RefineRefine with Stronger and Advanced

TL;DR

A schema validates structure: required fields, types, and allowed shapes. It does not prove that extracted values are factual, that a classification is justified, or that an identifier exists and belongs to the user. The application still validates domain rules and uses evaluation or human review where model judgment matters.

Full answer below

Practice focus

Whether structured outputs are used as a serialization guarantee rather than mistaken for a truth guarantee.

Answer first

Answer path

Core answer first, then checkpoints toward a stronger answer.

Start
Core answer
Start here. This is the direct spoken answer to practice first.
Overview
A constrained decoder can ensure that a category field is present and allowed; it cannot prove that the chosen category is supported by the input.
A schema validates structure: required fields, types, and allowed shapes. It does not prove that extracted values are factual, that a classification is justified, or that an identifier exists and belongs to the user. The application still validates domain rules and uses evaluation or human review where model judgment matters.
Checkpoint
Stronger answer
Optional upgrade with examples, trade-offs, and sharper phrasing.
A model can return a valid orderId string that names the wrong order. It can also choose a valid risk category based on weak evidence. I check identifiers against authoritative systems, validate cross-field rules, require evidence for consequential classifications, and route uncertain cases to a safe fallback instead of accepting the payload merely because it parsed.
Finish
Advanced answer
Optional deeper angle for production judgment and nuance.
I use separate validation layers rather than one generic success flag. Transport and schema checks establish shape, domain rules establish business validity, authorization protects access, and evidence checks support consequential claims. Each layer has distinct telemetry and recovery behavior. That lets us distinguish a schema regression from a model-quality regression or a data-access defect instead of combining them into a generic “bad AI response” metric.

Review signals

Use these to catch weak phrasing, shallow reasoning, and risky claims.

Common mistakesCandidateCommon mistakesMistakes that weaken the answer or signal shallow understanding.Open details

Details

Equating constrained decoding with factual correctness.
Trusting model-produced identifiers without lookup.
Using a confidence number as proof of accuracy.
Combining all validation failures into one retry path.

Red flagsCandidateRed flagsSignals that an interviewer may treat as risky.Open details

Details

A schema-valid payload can directly update protected state.
No evidence or authoritative lookup supports consequential fields.
The system cannot distinguish structural and semantic failures.

Follow-up prompts

Quick interviewer prompts inside this drill.

Follow-up questionsCandidateFollow-up questionsLikely next questions after the main answer.Open details

Details

Treat these as the next interviewer turn. Answer one out loud, then save the gap in your practice notes.

Which checks belong to domain code?
How would you validate extracted identifiers?
When should a human review the result?

Continue the section

Next in Prompting & Structured Output: Represent unknown values

Step 10 of 20

How should structured output represent unknown or missing information?

Practice route

Read core answer Review checklist Follow-up prompts Track progress

Finished this drill?

Practice freely. Sign in only when you want to save completion, confidence, notes, or bookmarks.

New to Aporeon? Create account

Drill details

Human editorial direction

Aporeon is shaped by Aleksandar Tomovski, a software developer with experience on both sides of technical interviews. Content is reviewed for accuracy, natural spoken delivery, useful depth, and honest trade-offs.

About the creator Editorial standards

Was this drill useful?

Help improve the interview library.

Found this answer useful?

Say thanks with a standalone, one-time $5 contribution. No account required.

No subscription or perks.Choose another amount

Opening drill

Concept questionMedium~10 minPrompting & Structured Output / 9 of 20

Interview question

Why can schema-valid model output still be wrong?

Separates structural validity from factual accuracy, evidence, authorization, and domain invariants.

1Say itAnswer out loud first

2CompareCompare with Core answer

3RefineRefine with Stronger and Advanced

TL;DR

Full answer below

Practice focus

Whether structured outputs are used as a serialization guarantee rather than mistaken for a truth guarantee.

Answer first

Answer path

Core answer first, then checkpoints toward a stronger answer.

Start
Core answer
Start here. This is the direct spoken answer to practice first.
Overview
A constrained decoder can ensure that a category field is present and allowed; it cannot prove that the chosen category is supported by the input.
A schema validates structure: required fields, types, and allowed shapes. It does not prove that extracted values are factual, that a classification is justified, or that an identifier exists and belongs to the user. The application still validates domain rules and uses evaluation or human review where model judgment matters.
Checkpoint
Stronger answer
Optional upgrade with examples, trade-offs, and sharper phrasing.
A model can return a valid orderId string that names the wrong order. It can also choose a valid risk category based on weak evidence. I check identifiers against authoritative systems, validate cross-field rules, require evidence for consequential classifications, and route uncertain cases to a safe fallback instead of accepting the payload merely because it parsed.
Finish
Advanced answer
Optional deeper angle for production judgment and nuance.
I use separate validation layers rather than one generic success flag. Transport and schema checks establish shape, domain rules establish business validity, authorization protects access, and evidence checks support consequential claims. Each layer has distinct telemetry and recovery behavior. That lets us distinguish a schema regression from a model-quality regression or a data-access defect instead of combining them into a generic “bad AI response” metric.

Review signals

Use these to catch weak phrasing, shallow reasoning, and risky claims.

Common mistakesCandidateCommon mistakesMistakes that weaken the answer or signal shallow understanding.Open details

Details

Equating constrained decoding with factual correctness.
Trusting model-produced identifiers without lookup.
Using a confidence number as proof of accuracy.
Combining all validation failures into one retry path.

Red flagsCandidateRed flagsSignals that an interviewer may treat as risky.Open details

Details

A schema-valid payload can directly update protected state.
No evidence or authoritative lookup supports consequential fields.
The system cannot distinguish structural and semantic failures.

Follow-up prompts

Quick interviewer prompts inside this drill.

Follow-up questionsCandidateFollow-up questionsLikely next questions after the main answer.Open details

Details

Treat these as the next interviewer turn. Answer one out loud, then save the gap in your practice notes.

Which checks belong to domain code?
How would you validate extracted identifiers?
When should a human review the result?

Continue the section

Next in Prompting & Structured Output: Represent unknown values

Step 10 of 20

How should structured output represent unknown or missing information?

Why can schema-valid model output still be wrong?

Practice focus

Answer path

Overview

Review signals

Follow-up prompts

Continue the section

Why can schema-valid model output still be wrong?

Practice focus

Answer path

Overview

Review signals

Follow-up prompts

Continue the section