Intelligent Routing and Failover
Understand server-side node selection, failover, and automatic retries in APIQIK.
APIQIK provides a unified endpoint over multiple inference nodes. Developers do not need to maintain node lists, health probes, or provider-switching logic.
Server-side flow
- Unified entry. Send requests to one APIQIK Base URL for account authentication and parameter validation.
- Node selection. The platform selects a route from nodes capable of serving the target model based on availability and health.
- Failure classification. Internal policy determines whether a temporary timeout or another retryable failure should trigger a switch.
- Automatic failover. The request is attempted on another available node; a successful response still uses the standard OpenAI-compatible structure.
Routing behavior contract
Retry counts and timeout budgets require platform confirmation. Actual values and behavior must be published before production reliance. Do not depend on a fixed attempt count or node-switching order in the meantime.
| Scenario | Expected principle | Platform confirmation required |
|---|---|---|
| Authentication, balance, or parameter error | Node switching cannot fix it; return a structured 4xx. | Error types excluded from internal retries. |
Connection failure, temporary overload, upstream 5xx | Another healthy node may be attempted before output is committed to the client. | Maximum attempts, backoff, and overall timeout budget. |
| Streaming output already sent | Seamless token-by-token continuation is not guaranteed; clients must handle partial output. | Interruption events, usage, and billing treatment. |
| Model or provider changes | Do not silently substitute a model with different semantics or capabilities. | Whether switching preserves the same model version. |
| Billing and idempotency | Internal attempts should not make a business request difficult to reconcile. | Failed-attempt billing, billing deduplication, and idempotency-key support. |
The same client and code → APIQIK health checks and routing → automatic selection among nodes A/B/C.
What developers do not need to implement
Provider health probes
No periodic requests to every provider to check node availability.
Node-level retries
No retry branches for individual inference nodes.
Connection adaptation
No separate authentication and request wrappers for different nodes.
Routing configuration
No hard-coded infrastructure node lists in business applications.
Client responsibilities
- Handle parameter, permission, balance, and other
4xxerrors that routing cannot resolve. - Set an overall deadline appropriate for the user experience.
- Use bounded business-level retries for safe requests when all platform routes fail.
- Record request ID, model, status, and duration to diagnose tail-latency issues.
- Handle retries and duplicate rendering carefully after partial streaming output.
Integration remains unchanged. Routing and failover happen on the server. Developers still call
/v1/chat/completionsregardless of the healthy node used underneath.