Rate Limits

Rate limits protect providers and your budget. They are evaluated atomically in one step before any provider is contacted, so a rate-limited request never costs you anything.

Scopes

All applicable scopes are checked; the first one exceeded produces the error. Windows are sliding (not fixed calendar minutes), so bursts at a boundary do not double your allowance.

tokens_per_minute is checked against the estimated token count (ceil(characters / 4) for messages, plus ~1,000 per image, plus max_tokens or the model default for output) since the real count is only known afterwards.

The 429 response

HTTP/1.1 429 Too Many Requests
Content-Type: application/json
Retry-After: 2
X-Request-Id: req_01J6ZK3M9PQR7S8T9V

{
  "error": {
    "type": "rate_limit_error",
    "code": "rate_limited",
    "message": "Rate limit exceeded for api_key (60 requests/minute)",
    "request_id": "req_01J6ZK3M9PQR7S8T9V",
    "retry_after": 2
  }
}

Headers

How to back off

1. On 429, sleep for Retry-After seconds (fall back to exponential backoff starting at 1 s with jitter if the header is missing), then retry the identical request.
2. Cap total retries (3–5) and surface the error to the caller afterwards.
3. Reduce concurrency when you see repeated concurrency_exceeded; a semaphore sized to the key's concurrent_requests prevents them entirely.
4. For throughput work, spread load across time rather than across keys — organization and model limits apply regardless of how many keys you create.

async function withBackoff<T>(fn: () => Promise<Response>, max = 4): Promise<Response> {
  for (let attempt = 0; ; attempt++) {
    const res = await fn();
    if (res.status !== 429 || attempt >= max) return res;
    const ra = Number(res.headers.get('retry-after'));
    const delay = Number.isFinite(ra) && ra > 0 ? ra * 1000 : Math.min(8000, 1000 * 2 ** attempt) * (0.5 + Math.random());
    await new Promise((r) => setTimeout(r, delay));
  }
}

import random, time, requests

def with_backoff(send, max_attempts=4):
    for attempt in range(max_attempts + 1):
        res = send()
        if res.status_code != 429 or attempt == max_attempts:
            return res
        ra = res.headers.get("Retry-After")
        delay = float(ra) if ra else min(8, 2 ** attempt) * (0.5 + random.random())
        time.sleep(delay)

Rate limits vs. spend limits

A 429 means slow down; a 402 means you cannot afford this request right now (balance, key budget or organization spending limit). Backing off does not fix a 402 — see API Keys for the difference.