Fallbacks

When a provider fails, the gateway first retries the same provider, then falls back to the next candidate from the routing decision. This page defines exactly when that happens.

Retryable vs non-retryable

Classification	Provider responses / conditions	Gateway behaviour
Retryable	HTTP `408`, `409` (provider-side), `425`, `429`, `500` (non-streaming only), `502`, `503`, `504`; connection refused/reset; DNS failure; timeout before the first byte	Retry with backoff, then fall back.
Non-retryable	HTTP `400`, `401`, `403`, `404`, `413`, `422`	Surface immediately — the request itself is at fault, or the provider's credentials/config are; another provider would fail the same way (or the operator must fix configuration). Returned as `502 provider_error` with the provider slug (never the upstream credentials).

A provider 429 is retryable but also counts toward that provider's circuit breaker (as half a failure), so a saturated provider is quickly de-prioritised for everyone.

Retry and backoff

Per provider, up to max_retries attempts (default 1 retry) with exponential backoff:

text

delay = min(cap, base × factor^attempt) ± 50 % jitter
base = 200 ms, factor = 2, cap = 2 s

If the provider sends Retry-After the gateway honours it, up to 5 seconds. After retries are exhausted the gateway moves to the next candidate, up to max_fallbacks (default 2). Every attempt — success or failure — is recorded to the provider's health record.

The whole sequence is bounded by the request timeout (default 120 s; streams use a 600 s idle-reset deadline). When the budget is spent the final error reflects the last failure:

Final error	When
`503 provider_unavailable`	Last attempt was a 5xx or connection failure.
`504 provider_timeout`	Last attempt timed out.
`429 provider_rate_limited`	Last attempt was a provider 429 (with `Retry-After`).
`502 provider_error`	Non-retryable provider error.
`503 no_provider_available`	No candidate was eligible to begin with.

error.provider names the last provider tried, and X-LLM-Routing-Reason shows what was considered. The final X-LLM-Provider header names the provider that actually answered.

The streaming rule

Retries and fallbacks on streaming requests are permitted only before the first content delta has been written to the client. The gateway holds back the SSE headers until it has a first chunk, so all of this happens invisibly and a total failure is still a normal JSON error with a proper status code.

After the first token has been delivered, a failure cannot be recovered by switching provider — the client already has partial output from a different model instance. Instead the stream ends with an error chunk (stream_interrupted or provider_disconnect) and [DONE]. Partial usage is charged. See Streaming.

The same principle applies to binary audio streams (/v1/audio/speech): fallback before the first byte only.

Idempotency for media

Media generation is expensive, and a provider may have accepted work even though the gateway saw an error. Therefore:

Video and music jobs are never retried at submit time beyond the same idempotent provider call. Once a job exists the worker retries polling, not generation. Submission requires Idempotency-Key, so your own retries are safe: the same key + same body returns the existing job.
Images and audio accept an optional Idempotency-Key. With it, a retried request that already completed returns the stored response (200, Idempotent-Replayed: true) and nothing is regenerated or charged again.

Idempotency semantics

Situation	Result
Same key, same body (method + path + body fingerprint), original finished	Stored result replayed with the original status (`200` images/audio, `202` jobs) and header `Idempotent-Replayed: true`.
Same key, same body, original still in flight	`409 idempotency_conflict` with `retry_after` (seconds) — retry later.
Same key, different body	`409 idempotency_conflict` — pick a new key.
Key longer than 255 characters	`400 validation_failed`.

Keys are scoped to your organization. Use UUIDs or a stable business identifier (order id, message id) so retries from any of your instances converge.

Client-side guidance

Treat 429, 502, 503, 504 and service_unavailable as retryable in your own client with exponential backoff; honour Retry-After when present.
Do not blindly retry 400, 401, 402, 403, 404, 409 (without retry_after), 413 — fix the request, key, budget or idempotency key.
For streams, if you receive an error chunk after partial output, decide per use case whether to re-issue the request (with a fresh conversation turn) or show the partial result.
For anything that costs real money (images, jobs), always send an Idempotency-Key.