Core answer
Start here. This is the direct spoken answer to practice first.
Overview
AI monitoring needs both ordinary service health and sampled evidence about response behavior.
I monitor standard service signals such as request rate, errors, latency, saturation, and dependency health, then add AI-specific signals tied to the task. Those include task success or sampled quality, unsupported or unsafe behavior, refusal and abstention rates, schema failures, tool failures, repair attempts, fallbacks, token usage, and estimated cost. Metrics are segmented by model, prompt, route, feature, and important user or task slices.