Concurrency: Running 25 API Calls Simultaneously

The Speed Problem

A single API call takes 3 to 15 seconds depending on the model, the prompt length, and the output length. A three-agent chain takes 9 to 45 seconds. Run 100 chains sequentially, and you are waiting 15 to 75 minutes. That is not a production bottleneck for content quality, but it is a bottleneck for your time. You are sitting idle, watching a progress bar, when you could be reviewing outputs.

Concurrency solves this. Instead of running one chain at a time, you run multiple chains simultaneously. The total processing time drops from N * time_per_chain to roughly N / concurrency_limit * time_per_chain.

How asyncio Works

Python's asyncio library lets you run multiple tasks concurrently without threads or processes. The core concept: when one task is waiting for an API response (which takes seconds), another task can start its API call. No task blocks the others.

Three pieces make this work:

The Semaphore Pattern

Running 100 API calls simultaneously will trigger rate limits and may overwhelm your system. A semaphore caps the number of concurrent operations.

The semaphore value depends on your API rate limits. If your provider allows 60 requests per minute and each chain makes 3 requests, a semaphore of 10 keeps you at approximately 30 active requests, well within limits.

Real Performance Numbers

Tested with a three-agent chain (research, write, edit) generating 1,000-word articles. Average chain time: 25 seconds.

At 25 concurrent chains, 100 articles generate in under 3 minutes. The bottleneck shifts entirely from generation to human review, which is where it should be.

Error Handling in Concurrent Batches

When one chain fails in a concurrent batch, it must not bring down the other chains. Wrap each chain in a try/except block. On failure, log the error and the failing row's ID. After the batch completes, retry only the failed rows.

async def safe_chain(topic, semaphore):
    async with semaphore:
        try:
            return await run_chain(topic)
        except Exception as e:
            log_error(topic, e)
            return {"id": topic.id, "status": "failed", "error": str(e)}

After batch completion, filter results for status "failed" and re-run only those rows. This gives you a complete batch with minimal wasted computation.

Connection Pooling

Each API call opens an HTTP connection. Opening 25 connections simultaneously and closing them after each call is wasteful. Connection pooling reuses connections across calls, reducing overhead. Python's aiohttp library handles this automatically with a session object.

Create one session at the start of the batch. Pass it to all chains. Close it when the batch completes. This single change can reduce per-call latency by 100 to 200 milliseconds, which compounds across hundreds of calls.

Further Reading

• Asynchronous LLM API Calls in Python, Unite.AI
• Mastering asyncio.gather for LLM Processing, Instructor
• LLM Parallel Processing in Practice, Dev.to
• Python Asyncio for LLM Concurrency, Newline