Start here. This is the direct spoken answer to practice first.
Why this question matters
Observability can become its own production problem when labels explode, logs flood storage, or sampling hides the one trace needed during an incident.
I would control volume with log levels, sampling, retention, and careful metric labels. Metrics should use low-cardinality dimensions like route template, status code, dependency name, region, or environment, not raw user ids, full URLs, or arbitrary search terms. Logs should keep important errors and decision points without dumping every request body.