Implement a model gateway with deadlines, budgets, fallback, and version evidence

• Validate routes at startup for capability, schema, region, and data-policy compatibility.
• Use one end-to-end deadline and derive per-attempt timeouts from remaining time.
• Retry only classified temporary failures and respect provider retry guidance within a small attempt budget.
• Enforce input/output token, estimated cost, and concurrency limits before and during execution.
• Do not automatically retry an ambiguous state-changing tool workflow.
• Return provider-neutral content plus model route, prompt, schema, latency, usage, and outcome versions.

1. Resolve an approved ordered route from capability and policy, rejecting incompatible configurations before traffic.
2. Reserve concurrency and cost capacity, then compute remaining time before every attempt.
3. Call the provider adapter with bounded input, output, and timeout settings.
4. Classify success, refusal, rate limit, temporary dependency failure, invalid output, and permanent policy failure.
5. Retry or move to a compatible fallback only when the policy and remaining budgets allow it.
6. Commit usage and emit one normalized result envelope without exposing provider-specific response objects.

async def execute(command: ModelCommand) -> ModelResult:
    route = routes.resolve(command.capability, command.policy)
    deadline = clock.monotonic() + command.timeout_seconds
    budget = await budgets.reserve(command.tenant_id, command.max_cost)

    async with concurrency.slot(command.tenant_id, command.capability):
        for target in route.targets:
            if not target.supports(command.requirements):
                continue
            for attempt in range(target.max_attempts):
                remaining = deadline - clock.monotonic()
                if remaining < target.minimum_attempt_seconds:
                    raise GatewayDeadlineExceeded()
                try:
                    raw = await target.adapter.generate(
                        command.payload,
                        timeout=min(remaining, target.per_attempt_timeout),
                        max_output_tokens=command.max_output_tokens,
                    )
                    result = normalize_and_validate(raw, command.requirements)
                    await budget.commit(result.usage)
                    return result.with_route_evidence(target, attempt)
                except TemporaryProviderError as error:
                    if not retry_policy.allows(error, attempt, remaining):
                        break
                    await clock.sleep(retry_policy.delay(error, attempt))
                except (Refusal, PolicyViolation, InvalidRequest):
                    raise
    raise NoCompatibleModelSucceeded()

Routing is O(number of configured targets and bounded attempts). Provider calls dominate latency, while explicit limits cap simultaneous work, generated tokens, and cost exposure.

• A 429 retry-after value exceeds the caller's remaining deadline.
• The primary supports a schema feature that the fallback does not.
• The cost estimate fits, but actual output approaches the reserved limit.
• The client cancels while the provider request is still in flight.
• A regional outage leaves only a target that violates data-residency policy.

• Advance a fake clock and assert no attempt begins without enough remaining time.
• Return a permanent invalid request and assert neither retry nor fallback masks it.
• Configure an incompatible fallback and reject configuration at startup.
• Saturate one tenant's concurrency without blocking an unrelated tenant's reserved capacity.
• Verify telemetry contains versions and usage but no prompt or generated content.

• Giving each retry a fresh full timeout violates the caller's end-to-end latency budget.
• Falling back to any available model can silently break schemas, regions, safety, or quality.
• Retrying provider-specific exceptions in every feature duplicates policy and creates inconsistent behavior.
• Estimating cost only after completion cannot protect a hard tenant budget.
• Returning raw provider objects couples application code to unstable transport details.