How would you apply timeouts, retries, and cancellation to model calls?

Uses deadline budgets, transient-only bounded retries, jitter, idempotency, and propagated cancellation without retry storms.

1Say itAnswer out loud first

2CompareCompare with Core answer

3RefineRefine with Stronger and Advanced

TL;DR

I give the user operation an overall deadline and allocate smaller time budgets to model, retrieval, and tool calls. Retries are limited to transient failures such as throttling or selected server errors, use exponential backoff with jitter, and stop when the remaining deadline cannot support another attempt. Cancellation from the client or workflow propagates through the application even if a provider cannot always stop work already accepted.

Full answer below

Practice focus

Distributed failure handling, retry safety, deadlines, and resource control.

Answer first

Answer path

Core answer first, then checkpoints toward a stronger answer.

Start
Core answer
Start here. This is the direct spoken answer to practice first.
Overview
Model calls are remote work with variable duration, cost, and ambiguous outcomes.
I give the user operation an overall deadline and allocate smaller time budgets to model, retrieval, and tool calls. Retries are limited to transient failures such as throttling or selected server errors, use exponential backoff with jitter, and stop when the remaining deadline cannot support another attempt. Cancellation from the client or workflow propagates through the application even if a provider cannot always stop work already accepted.
Checkpoint
Stronger answer
Optional upgrade with examples, trade-offs, and sharper phrasing.
I do not retry invalid requests, policy refusals, semantic failures, or every timeout blindly. Streaming and state-changing tool paths need special care because the first attempt may have produced partial output or completed a side effect. Stable operation IDs, idempotent tool calls, and recorded provider request IDs make ambiguous outcomes diagnosable.

Review signals

Use these to catch weak phrasing, shallow reasoning, and risky claims.

Common mistakesCandidateCommon mistakesMistakes that weaken the answer or signal shallow understanding.Open details

Details

Retrying every exception with the same delay.
Giving each dependency the full user timeout.
Assuming cancellation means the provider did no work.

How would you apply timeouts, retries, and cancellation to model calls?

Practice focus

Answer path

Overview

Review signals

Follow-up prompts

Continue practice

Next in path

Tech stacks

Skill areas

Human editorial direction

Was this drill useful?

Found this answer useful?