Fetch URLs concurrently with a bound, timeout, and structured cleanup

Implement fetch_all(urls, limit, timeout_seconds) using one httpx.AsyncClient. Run at most limit requests at once, keep results in input order, fail the operation when any request fails, and ensure all sibling tasks and the client are cleaned up. Do not create every connection without a concurrency bound.

urls=[A,B,C], limit=2 -> at most two client.get calls active
B fails              -> siblings are cancelled and awaited
results              -> returned in A,B,C input order

• Reject an empty or non-positive concurrency limit.
• Use one client for the whole operation.
• Apply a deadline to each request.
• Use structured task ownership rather than detached tasks.
• Preserve input order without serializing completion.

1. Allocate a fixed result list and one semaphore.
2. Create one coroutine per indexed URL inside a TaskGroup.
3. Acquire the semaphore only around the bounded remote call.
4. Use asyncio.timeout and raise_for_status inside the owned task.
5. Let TaskGroup cancel and await siblings on failure, then close the client context.

import asyncio
import httpx

async def fetch_all(
    urls: list[str], limit: int, timeout_seconds: float
) -> list[bytes]:
    if limit <= 0:
        raise ValueError("limit must be positive")

    semaphore = asyncio.Semaphore(limit)
    results: list[bytes | None] = [None] * len(urls)

    async with httpx.AsyncClient() as client:
        async def fetch(index: int, url: str) -> None:
            async with semaphore:
                async with asyncio.timeout(timeout_seconds):
                    response = await client.get(url)
                    response.raise_for_status()
                    results[index] = response.content

        async with asyncio.TaskGroup() as group:
            for index, url in enumerate(urls):
                group.create_task(fetch(index, url))

    return [value for value in results if value is not None]

The work is O(n) requests with O(n) task and result bookkeeping, while active remote calls are bounded by limit.

• An empty URL list returns an empty result.
• A timeout produces an exception group through TaskGroup.
• Duplicate URLs remain distinct indexed requests.
• A caller cancellation must still close the client and await children.

• Instrument active calls and assert the peak never exceeds the limit.
• Complete responses out of order and verify returned ordering.
• Fail one request and verify no child remains running.
• Cancel the parent and verify the client context exits.

• gather over unbounded coroutines can overwhelm sockets and the dependency.
• Creating one client per URL defeats pooling.
• Detached create_task calls can outlive the operation.
• Appending results records completion order rather than input order.