Instrument an AI request trace without leaking prompts or customer data

Instrument a request that performs retrieval, generation, validation, and optional repair. Operators must locate slow or failing stages and compare versions, but prompts, retrieved text, generated content, user IDs, and document titles are forbidden from default telemetry. Add spans, metrics, and structured outcomes that support latency, cost, refusal, validation, and quality investigations.

Trace: ai.answer -> ai.retrieve -> ai.generate -> ai.validate -> ai.repair
Safe attributes: operation, route ID, model version, prompt version, schema version, token counts, result code
Forbidden by default: prompt text, response text, chunk text, email, tenant name, raw document ID.

• Create duration-bearing spans for major operations and structured events or logs for point-in-time outcomes.
• Use stable low-cardinality names and version identifiers.
• Hash or tokenize identifiers only when there is an approved operational need and bounded retention.
• Record token counts and estimated cost without recording content.
• Separate default telemetry from tightly controlled diagnostic capture.
• Preserve trace context across asynchronous work and provider adapters.

1. Write the incident questions telemetry must answer before choosing attributes.
2. Define one parent request span and child spans around retrieval, generation, validation, and repair boundaries.
3. Attach operation, route, version, outcome, token, and retry attributes with bounded vocabularies.
4. Emit metrics for latency, failure/refusal/repair rates, tokens, cost, and quality proxies by safe dimensions.
5. Propagate trace context through jobs and adapters, and correlate structured exception or state-change events.
6. Test the exporter payload for forbidden fields and document the separate break-glass diagnostic path.

async def answer(request: AnswerRequest) -> AnswerResult:
    with tracer.start_as_current_span("ai.answer") as root:
        root.set_attributes({
            "gen_ai.operation.name": "answer",
            "app.ai.route.id": request.route_id,
            "app.ai.prompt.version": PROMPT_VERSION,
            "app.ai.schema.version": SCHEMA_VERSION,
        })
        with tracer.start_as_current_span("ai.retrieve") as span:
            chunks = await retrieve(request.safe_scope, request.query)
            span.set_attribute("app.ai.retrieved.count", len(chunks))

        with tracer.start_as_current_span("ai.generate") as span:
            generated = await gateway.generate(request.query, chunks)
            span.set_attributes({
                "gen_ai.request.model": generated.model_id,
                "gen_ai.usage.input_tokens": generated.input_tokens,
                "gen_ai.usage.output_tokens": generated.output_tokens,
                "app.ai.outcome": generated.safe_outcome,
            })

        with tracer.start_as_current_span("ai.validate") as span:
            result = validate(generated, chunks)
            span.set_attribute("app.ai.validation.result", result.safe_code)
        return result

# Exporter policy rejects prompt, response, chunk text, email, and raw customer IDs.

Instrumentation adds O(number of workflow stages) records and constant per-stage metadata. Export volume and high-cardinality dimensions, not algorithmic computation, are the main scaling risks.

• A provider SDK automatically captures prompts unless content capture is disabled.
• A repair call creates another model span and must remain connected to the original request.
• An asynchronous evaluator runs after the request and needs a correlation ID without inheriting sensitive baggage.
• Error messages contain fragments of provider payloads.
• Sampling drops common successful traces but must retain enough version coverage for comparisons.

• Inspect serialized exporter payloads and fail on known prompt, output, email, or document text.
• Trigger a repair and verify the trace shows both attempts under one parent.
• Run work through a queue and verify trace or link correlation survives process boundaries.
• Generate thousands of user IDs and assert they are not metric label values.
• Query failures by prompt and model version and verify the recorded attributes support the investigation.

• Capturing all prompts and responses makes debugging easy by turning telemetry into a sensitive-data store.
• One giant request span cannot show whether retrieval, generation, or validation caused latency.
• User-controlled strings in span names or metric labels create cardinality and cost explosions.
• Token totals without route and version context cannot explain a release regression.
• Hashing every identifier does not automatically make the data anonymous or necessary.