An AI feature became slower and more expensive. How would you diagnose it?

Diagnoses an AI regression by decomposing request stages, configuration changes, token growth, retries, routing, tools, retrieval, caching, and provider behavior.

1Say itAnswer out loud first

2CompareCompare with Core answer

3RefineRefine with Stronger and Advanced

TL;DR

I confirm the time window and affected feature, tenant, model, and request class, then split latency and cost by stage: context construction, retrieval, model calls, tool calls, validation, retries, and queueing. I compare the deployed configuration and input distributions with the previous baseline. Common causes include longer history or retrieved text, a model or routing change, more generated tokens, retry storms, cache misses, slower tools, or provider throttling.

Full answer below

Practice focus

Evidence-led production diagnosis across both conventional service signals and AI-specific usage and workflow dimensions.

Answer first

Answer path

Core answer first, then checkpoints toward a stronger answer.

Start
Core answer
Start here. This is the direct spoken answer to practice first.
Overview
The investigation should locate which stage and request slice changed before attempting prompt or model optimizations.
I confirm the time window and affected feature, tenant, model, and request class, then split latency and cost by stage: context construction, retrieval, model calls, tool calls, validation, retries, and queueing. I compare the deployed configuration and input distributions with the previous baseline. Common causes include longer history or retrieved text, a model or routing change, more generated tokens, retry storms, cache misses, slower tools, or provider throttling.
Checkpoint
Stronger answer
Optional upgrade with examples, trade-offs, and sharper phrasing.
I use traces to follow representative slow and expensive requests and correlate provider request IDs, token counts, finish reasons, model version, retry and fallback paths, and dependency timings. p95 and p99 matter because averages can hide one costly slice. I check whether quality also changed: an optimization that shortened output may have triggered more repairs, while a retrieval change may have increased both context size and unsupported answers.

Review signals

Use these to catch weak phrasing, shallow reasoning, and risky claims.

Common mistakesCandidateCommon mistakesMistakes that weaken the answer or signal shallow understanding.Open details

Details

Blaming the provider before decomposing the workflow.
Looking only at average latency or aggregate spend.
Ignoring retries, repairs, tool loops, and cache misses.
Logging complete sensitive prompts to gain visibility.

An AI feature became slower and more expensive. How would you diagnose it?

Practice focus

Answer path

Overview

Review signals

Follow-up prompts

Continue practice

Next in path

Technologies

Tech stacks

Skill areas

Human editorial direction

Was this drill useful?

Found this answer useful?