Routing

Every model in the catalog can be served by one or more providers. For each request the routing engine builds the list of eligible providers, ranks them with a strategy, then executes with retry and fallback down that list. You never address a provider directly — you ask for a model and, optionally, express a preference.

How candidates are chosen

Providers are filtered before ranking. A provider is eligible for a request when it:

is enabled and serves the requested model (with a provider-specific model mapping),
supports the required capabilities (e.g. streaming, tools, vision, json_mode) for this request,
is not excluded by your organization's routing policy or the request's routing.providers,
has a closed or half-open circuit — providers whose circuit breaker is open after consecutive failures are skipped until a probe succeeds.

If no provider survives filtering the request fails with 503 no_provider_available. Each rejected candidate keeps its reason; the dashboard's request detail shows the full candidate list.

Strategies

Strategy	Ranking	Use when
`priority_cheapest` (default)	Provider priority (asc), then estimated cost (asc), then observed latency (asc)	You want the operator's preferred providers, cheapest first among equals.
`cheapest`	Estimated cost (asc), then priority	Cost matters most; latency and preference are secondary.
`fastest`	Exponentially-weighted average latency (asc; unknown latency sorts last), then cost	Interactive UIs where time-to-first-token dominates.
`priority`	Priority (asc), then the policy's `provider_order`, then cost	Strict operator-defined ordering.
`highest_availability`	Recent success rate (desc), then latency, then cost	Batch or critical paths where completion matters more than price.
`smart`	Not yet available. Accepted for forward compatibility; the engine falls back to `priority_cheapest` and records reason `smart_fallback`.	—

Cost used for ranking is the estimated cost of this request (see Pricing) using each candidate's price sheet. Latency and success rate come from a per-provider health record updated after every attempt.

Per-request override

Add the routing object to any inference request (chat, completions, responses, embeddings, images, speech, video, music). It is validated, applied, then stripped before the request reaches a provider.

json

{
  "model": "meta/llama-3.1-70b-instruct",
  "messages": [{ "role": "user", "content": "Summarise this…" }],
  "routing": {
    "strategy": "fastest",
    "providers": ["groq", "fireworks"]
  }
}

Field	Effect
`strategy`	Replaces the organization's default strategy for this request only.
`providers`	Whitelist of provider slugs (max 10). Only these are eligible; order is used as a tie-breaker for `priority`. Slugs that do not serve the model are ignored.

Provider slugs are shown in X-LLM-Provider on responses and in the dashboard's Routing page. Overrides cannot bypass disabled providers, open circuits, or your key's allowed_models scope.

Organization policy

Your organization's default strategy, provider ordering, exclusions, and the max_retries / max_fallbacks budgets are configured in the dashboard under Routing. Per-request overrides take precedence over the organization policy, which in turn takes precedence over the platform default.

How the decision is exposed

Response headers

Header	Content
`X-LLM-Provider`	Provider that produced the final response (after any fallback).
`X-LLM-Model`	Resolved gateway model id.
`X-LLM-Routing-Reason`	Compact key=value summary of the decision.

http

X-LLM-Routing-Reason: strategy=priority_cheapest selected=groq priority=10 cost_micro=42 healthy=true fallbacks=openai

Key	Meaning
`strategy`	Strategy actually applied (after `smart_fallback` if relevant).
`selected`	Top-ranked provider at decision time.
`priority`	Its configured priority.
`cost_micro`	Its estimated cost for this request.
`healthy`	Whether its circuit was closed.
`fallbacks`	Comma-separated remaining candidates in order.

If a fallback occurred, X-LLM-Provider will differ from selected — that is the quickest way to spot fallbacks in your logs.

Response body

provider appears at the top level of every response object and every stream chunk.

Dashboard

The Requests explorer shows, per request, every candidate with its eligibility result, the ranking, each attempt's outcome and latency, and the final provider — a visual version of the header above.

Health and circuit breaking

Each provider attempt records success, error, timeout or 429 with its latency. After a configurable number of consecutive failures a provider's circuit opens and it is skipped for a cool-down period; the gateway then admits a few probe requests (half-open) and closes the circuit on success. 429s count as half a failure. Background probes every 60 seconds can close an open circuit early without spending your credits. The derived state (healthy, degraded, unhealthy, disabled) per provider and model is visible in the dashboard.