LLM Gatewaydocs

Streaming

Set "stream": true on /v1/chat/completions, /v1/completions or /v1/responses to receive tokens as they are generated. The gateway speaks standard OpenAI Server-Sent Events, so any SSE-capable OpenAI client works unchanged.

Wire format

Response headers:

http
HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive
X-Accel-Buffering: no
X-Request-Id: req_01J6ZK3M9PQR7S8T9V
X-LLM-Provider: groq
X-LLM-Model: meta/llama-3.1-70b-instruct
X-LLM-Routing-Reason: strategy=fastest selected=groq …
X-LLM-Cost-Micro: 120

Body — one data: line per chunk, terminated by a blank line; data: [DONE] ends the stream:

text
: ping

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1725446400,"model":"meta/llama-3.1-70b-instruct","provider":"groq","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1725446400,"model":"meta/llama-3.1-70b-instruct","provider":"groq","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1725446400,"model":"meta/llama-3.1-70b-instruct","provider":"groq","choices":[{"index":0,"delta":{"content":" there"},"finish_reason":null}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1725446400,"model":"meta/llama-3.1-70b-instruct","provider":"groq","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1725446400,"model":"meta/llama-3.1-70b-instruct","provider":"groq","choices":[],"usage":{"prompt_tokens":9,"completion_tokens":2,"total_tokens":11,"x_llm_cost_micro":2}}

data: [DONE]

Rules your parser must handle:

  1. Comments. Lines starting with : are SSE comments. While the gateway waits for the first token it sends : ping every 15 seconds so idle proxies keep the connection open. Ignore them.
  2. Chunk shape. Every data: payload except [DONE] is a chat.completion.chunk (or an error, below). provider is present on every chunk.
  3. Final usage chunk. The last data chunk before [DONE] has an empty choices array and a usage object. It is sent always, regardless of stream_options.include_usage. usage.x_llm_cost_micro is the charged amount; x_llm_usage_estimated: true means the provider did not report tokens and the gateway estimated output tokens from text length.
  4. Terminator. data: [DONE] is always the last event, including after an error chunk.

Tool calls in streams

Tool calls stream as partial delta.tool_calls[] entries with index, id (first fragment only), function.name and incremental function.arguments strings. Concatenate arguments per index until finish_reason: "tool_calls".

Errors during a stream

Two situations, distinguished by whether any content has been sent:

Before the first token

If the provider fails before any content delta has been written, the gateway retries and falls back to other providers exactly as for non-streaming requests (see Fallbacks). SSE headers are deferred until the first chunk, so if every attempt fails you receive an ordinary JSON error with the proper HTTP status — not a 200 with an error chunk:

http
HTTP/1.1 503 Service Unavailable
Content-Type: application/json
X-Request-Id: req_…

{"error":{"type":"provider_error","code":"provider_unavailable","message":"All providers failed","request_id":"req_…","provider":"groq"}}

After the first token

Once content has been delivered the stream cannot be restarted (the client already holds partial output). A provider failure or disconnect at this point is reported as an error chunk followed by [DONE]:

text
data: {"error":{"type":"provider_error","code":"stream_interrupted","message":"provider connection lost","request_id":"req_01J6ZK3M9PQR7S8T9V","provider":"groq"}}

data: [DONE]
codeMeaning
stream_interruptedProvider returned an error or timed out mid-stream.
provider_disconnectProvider closed the connection without a finish_reason.

The HTTP status is already 200; detect these by checking for an error key in each parsed chunk. Partial usage is recorded and charged; the request appears with status partial in the dashboard.

Client disconnect

If you close the connection, the gateway cancels the upstream provider request immediately, records the tokens generated so far (estimated when the provider sent no usage), and charges only that partial usage. The request is recorded with status client_disconnect.

Timeouts

TimeoutDefaultBehaviour
Idle between chunks60 sExceeded → stream_interrupted error chunk.
Overall stream deadline600 s, reset on activityExceeded → stream_interrupted error chunk.
Time to first byteGoverned by the provider timeout in the routing policyExceeded before first token → retried/fallback; final failure 504 provider_timeout.

Parsing SSE by hand

If you are not using an SDK, read the body incrementally, split on blank lines, and take the text after data: :

typescript
const res = await fetch('https://api.smartapihub.com/v1/chat/completions', {
  method: 'POST',
  headers: { Authorization: `Bearer ${process.env.LLM_API_KEY}`, 'Content-Type': 'application/json' },
  body: JSON.stringify({ model: 'openai/gpt-4o-mini', stream: true, messages: [{ role: 'user', content: 'Hi' }] }),
});
if (!res.ok) throw new Error((await res.json()).error.message);

const reader = res.body!.getReader();
const decoder = new TextDecoder();
let buffer = '';
for (;;) {
  const { value, done } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });
  let idx: number;
  while ((idx = buffer.indexOf('\n\n')) >= 0) {
    const event = buffer.slice(0, idx);
    buffer = buffer.slice(idx + 2);
    for (const line of event.split('\n')) {
      if (!line.startsWith('data: ')) continue; // skips ": ping" comments
      const data = line.slice(6);
      if (data === '[DONE]') break;
      const chunk = JSON.parse(data);
      if (chunk.error) throw new Error(`${chunk.error.code}: ${chunk.error.message}`);
      if (chunk.usage) console.log('cost µ$', chunk.usage.x_llm_cost_micro);
      process.stdout.write(chunk.choices[0]?.delta?.content ?? '');
    }
  }
}

A Python equivalent using requests is on the SDK Examples page.

ResponsesEmbeddings