Implement an idempotent retryable Celery provider task

• Pass only the durable operation ID in the Celery message.
• Reload current state in a task-owned database session.
• Reuse the stored provider idempotency key on every attempt.
• Retry only classified temporary failures with a maximum attempt count.
• Commit provider outcome before reporting task success.

1. Load the operation and return when it is already terminal.
2. Atomically claim pending work or detect another active owner.
3. Call the provider with explicit connect and response timeouts.
4. Persist success using the stable provider result.
5. On a temporary error, record attempt context and call self.retry; reconcile ambiguous exhaustion.

from celery import shared_task

@shared_task(bind=True, acks_late=True, max_retries=5)
def charge_payment(self, operation_id: str) -> None:
    with session_factory() as session:
        operation = load_operation_for_update(session, operation_id)
        if operation.status == "succeeded":
            return
        if operation.status == "failed":
            raise PermanentPaymentFailure(operation.failure_code)
        provider_key = operation.provider_idempotency_key

    try:
        result = provider.charge(
            amount=operation.amount,
            idempotency_key=provider_key,
            timeout=(2.0, 8.0),
        )
    except TemporaryProviderError as exc:
        record_retry_attempt(operation_id, type(exc).__name__)
        raise self.retry(exc=exc, countdown=min(60, 2 ** self.request.retries))
    except PermanentProviderError as exc:
        mark_failed(operation_id, exc.safe_code)
        return

    mark_succeeded_once(operation_id, result.charge_id)

Each attempt performs bounded indexed state reads/writes plus one provider call. Retry cost is bounded by the configured attempt budget.

• The provider can accept a charge and the client can still time out.
• A worker can die after provider success but before the local commit.
• Two deliveries can overlap unless the claim or state transition is atomic.
• Retry exhaustion with an ambiguous provider result requires reconciliation.

• Return success, then redeliver the same task and verify no new charge.
• Timeout after simulated provider acceptance and verify the same key is reused.
• Return a permanent decline and verify no retry is scheduled.
• Run concurrent deliveries and assert one terminal operation outcome.

• Generating a fresh provider key on retry can create duplicate charges.
• Retrying every exception wastes capacity and repeats permanent failures.
• Keeping a request-scoped session in task arguments is invalid ownership.
• Treating task success as proof of exactly-once delivery ignores redelivery.