Guides
Timeouts, Retries, and Concurrency
Set connection and overall timeouts, use exponential backoff, and manage concurrency with bounded queues.
The platform provides node-level failover, but clients still need explicit timeouts, bounded retries, and concurrency controls to protect their own services.
Timeout design
| Timeout | Scope | Recommendation |
|---|---|---|
| Connection | Establishing TCP/TLS | Keep it short so unreachable networks do not occupy resources indefinitely. |
| Time to first token | Sending the request to receiving the first streamed chunk | Match model reasoning behavior and interaction requirements. |
| Read | Waiting between chunks | Allow more time for long outputs and reasoning models. |
| Overall business deadline | From entering the application to final response | Include queueing, platform routing, and generation time. |
Python client configuration
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["APIQIK_API_KEY"],
base_url="https://api.apiqik.com/v1",
timeout=60.0,
max_retries=2,
)SDK retries and business retries compound. Calculate the worst-case attempt count and wait time before setting the overall business deadline.
Retry principles
- Retry only transient errors, such as some
429,500,502,503,504, and network failures. - Do not retry invalid parameters, authentication failures, or insufficient balance; fix the cause first.
- Use exponential backoff with jitter to avoid synchronized retries.
- Set maximum attempts and a total time budget; fail promptly with a clear message when exhausted.
- Establish idempotency before enabling automatic retries for operations with external side effects.
Concurrency control
Business requests → bounded queue and concurrency semaphore → APIQIK.
Use a bounded queue to limit in-flight requests. Reject or degrade when the queue is full rather than creating tasks until memory is exhausted. Reduce the sending rate after 429 and prioritize retry guidance in the response when available.
Degradation strategies
- Disable unnecessary long answers, deep reasoning, or high-cost features.
- Queue low-priority work to protect critical interactive traffic.
- Switch to a faster or less expensive model only after evaluation.
- Serve validated cached answers where appropriate and show when they were updated.