Cookbook
High-Concurrency Practices
Build stable concurrent API services with async clients, connection reuse, semaphores, and bounded queues.
High concurrency does not mean creating unlimited requests. Reuse clients, bound in-flight work, apply backpressure, and monitor tail latency.
Async concurrency example
import asyncio
import os
from openai import AsyncOpenAI
client = AsyncOpenAI(
api_key=os.environ["APIQIK_API_KEY"],
base_url="https://api.apiqik.com/v1",
timeout=60.0,
max_retries=2,
)
semaphore = asyncio.Semaphore(20)
async def ask(question):
async with semaphore:
response = await client.chat.completions.create(
model="qwen-turbo",
messages=[{"role": "user", "content": question}],
max_tokens=300,
)
return response.choices[0].message.content
async def main():
questions = [
"Explain API gateways",
"Explain intelligent routing",
"Explain exponential backoff",
]
results = await asyncio.gather(
*(ask(question) for question in questions),
return_exceptions=True,
)
for result in results:
print(result)
asyncio.run(main())Capacity protection
| Mechanism | Purpose | Failure strategy |
|---|---|---|
| Concurrency semaphore | Limit simultaneous upstream requests. | Bound waiting and return a busy response after timeout. |
| Bounded queue | Absorb short traffic bursts. | Reject low-priority work when full. |
| Rate limiting | Control sending rates by user, tenant, or feature. | Return clear retry guidance or a degraded result. |
| Circuit breaker | Reduce futile calls during sustained failure. | Fail fast briefly, then restore traffic gradually after recovery probes. |
| Cache | Reuse stable, repeated results when caching is allowed. | Define versioning, expiration, and data isolation. |
Connection and client lifecycle
- Reuse the client throughout the process lifecycle rather than constructing one per request.
- Size concurrency from instance memory, file descriptors, and business latency, not theoretical QPS alone.
- Separate large and small requests so long generations do not block interactive work.
- Increase load gradually and sustain tests long enough to observe P99 and recovery.
- Platform failover does not replace your own ingress rate limiting and resource isolation.