Streaming responses

Streaming changes response delivery, not the Gateway’s authentication, model-readiness, policy, and routing gates. Use it when incremental text matters and your client can parse server-sent events without mistaking transport completion for model success.

Client cancellation, resume, and reconnect guarantees are not defined by the supplied server implementation.

Streaming clients should model the response as a protocol with state, not as a sequence of arbitrary text fragments. The role event establishes the assistant message, delta events extend its content, the final event carries usage, and [DONE] closes the stream. An error event changes the outcome even if deltas were already received. Capturing request and trace headers before consuming the body is therefore essential: a partial stream may need investigation after the connection has ended and no normal JSON error body is available.

When to stream

Streaming is useful when the client should render partial output before the provider has finished. It changes delivery, not the routing contract: authentication, model readiness, budget, provider selection, policy, credentials, aliases, and timeouts are still evaluated before or during the provider attempt.

Use non-streaming first when bringing up a new integration. A successful JSON response gives a simpler baseline for authentication and route metadata. Add streaming after that baseline works so client parsing and partial-delivery behavior can be tested independently.

Choose streaming when the caller can use partial text before the complete response is available.

Keep non-streaming for simpler integrations that need one atomic JSON object.

Streaming does not bypass budget checks, provider eligibility, credentials, or fallback policy.

Stream when incremental text improves the user experience or reduces perceived latency, not because it changes provider eligibility or bypasses Gateway validation. The same key, model, budget, routing, and policy checks occur before events are delivered.

A stream can fail after provider selection, so routing success and complete client delivery are separate outcomes.

The final chunk carries usage information before [DONE]. Streaming and non-streaming requests use the same routing controls and provider eligibility checks.

A partial provider failure emits an error chunk and still terminates with [DONE]. Set stream: true on the verified chat-completions request.

Request configuration

Set stream to true on the verified chat-completions request. The remaining body follows the same supported field set as non-streaming calls. The server returns text/event-stream and supplies stream, request, and trace headers that should be retained with the event log.

Do not add reconnect tokens, cursor fields, or resume parameters. The inspected handler does not define them. Provider timeout values can be supplied through Gateway options where configured, but no universal stream timeout is documented.

Use the same POST path, Bearer key, model, and messages as a non-streaming call.

Generation and gateway routing fields remain limited to those implemented by the chat request interface.

Set stream to true and keep the remaining request within the verified chat contract. The client should be prepared for SSE framing, response headers that carry identifiers, and an error event that arrives after some text has already been emitted.

The response content type is text/event-stream and includes stream, request, and trace headers.

The first chunk establishes the assistant role before content deltas arrive.

Event sequence

A successful stream begins with an assistant-role chunk, continues with content deltas, and ends with a final chunk that carries completion and usage information before the [DONE] sentinel. Clients should parse complete SSE events rather than individual network packets. If the provider fails after output has started, the handler can emit an error event and still close the stream; receiving [DONE] does not erase the preceding failure.

The stream begins with a chunk that establishes the assistant role. Content then arrives through delta chunks. The final normal chunk includes usage, after which the server emits [DONE]. A client should process these stages in order rather than concatenate every event as ordinary text.

A provider can fail after the response has started. In that case, the handler emits an error chunk and still sends [DONE]. The sentinel therefore marks transport termination; it does not by itself prove that generation completed successfully. Persist the error state separately from accumulated text.

The response content type is text/event-stream.

The route emits an opening role chunk, zero or more delta chunks, a final chunk containing usage, and a terminal [DONE] sentinel.

Request, trace, and stream identifiers are returned in headers for correlation.

SSE request

{
  "model": "YOUR_RUNNABLE_MODEL_ID",
  "messages": [
    {"role": "user", "content": "Stream a short response."}
  ],
  "stream": true
}

A compliant consumer reads role, delta, and final chunks in order and stops after the [DONE] sentinel.

The event sequence has recognizable phases: an opening assistant-role chunk, zero or more content deltas, a final chunk that includes usage, and [DONE]. Clients should not treat every event as user-visible text.

The client must distinguish content deltas, the final usage-bearing chunk, the terminal sentinel, and any provider-stream error.

Do not infer reconnect, replay, or resume behavior from the server-sent event format when those guarantees are not implemented.

The streaming response uses text/event-stream and returns stream, request, and trace identifiers in headers.

Client handling

Backpressure and rendering policy belong to the client. The verified server behavior describes event order but does not prescribe how often the interface redraws, how deltas are buffered, or whether partial text is retained after failure. Choose those behaviors explicitly and keep the final request status authoritative.

If the client parser encounters malformed framing, stop treating subsequent bytes as trusted content. Retain the raw event boundary, request and trace IDs, and last successfully parsed event for diagnosis.

A robust client separates four concerns: SSE framing, content accumulation, route metadata, and final status. Parse each event boundary, append only content deltas, capture usage from the final chunk, and surface an error event as a failed result. Preserve request and trace identifiers so the partial stream can be correlated with provider attempts and logs.

Stream persistence is best-effort. An application should not infer that every partial delta has been durably stored simply because it was displayed. If the client needs its own transcript, it should build that transcript from the received events while retaining the final success or failure state.

Parse each data event in order and accumulate only the supported delta content.

Capture the final usage-bearing event separately from generated text.

Treat Ethen metadata and headers as operational context and preserve them with client-side diagnostics.

Client handling needs two accumulators: one for generated content and another for operational metadata and usage. Preserve request and trace headers before reading the body so a partial failure can still be investigated.

A streaming client needs a small state machine. Accept the opening role event without appending visible text, concatenate subsequent delta content in arrival order, and hold the request open until final usage or error information has been processed. [DONE] terminates the protocol; it is not a content token. The response headers and Ethen identifiers should be captured before the body is consumed so later diagnostics can correlate the stream with logs.

Partial output does not convert an error into success. If a provider fails after emitting text, the handler can send an error chunk and still close the event stream with [DONE]. The caller should mark the request failed, retain any partial text only according to application policy, and avoid presenting incomplete output as a normal completion. The supplied source does not define replay, cursor-based resume, or automatic reconnection, so reconnecting creates a new request unless a separate application-level design says otherwise.

Cancellation

A caller-side disconnect is not documented as a universal provider cancellation signal. Treat abandoned client connections and provider termination as separate concerns.

No public resume or reconnection guarantee is documented, and cancellation propagation can vary by provider adapter. Document only the observable result in the caller: the client stopped reading, the connection closed, or an error event arrived.

When cancellation behavior matters operationally, capture the request and trace IDs, the last event received, and whether a usage-bearing final chunk or [DONE] was observed. Those details distinguish a client-side stop from a provider-stream failure without asserting unsupported provider semantics.

Durable resume tokens and replay from a disconnected offset are not defined.

A client may stop reading its connection, but server-side cancellation semantics are not guaranteed by this bundle.

Design consumers so an interrupted stream can be reported as incomplete rather than silently promoted to a complete answer.

The source does not define resumable streams or a server-side cancellation protocol. Closing the client connection may stop delivery, but documentation should not promise provider cancellation, replay, or continuation from the last delta.

The first SSE chunk establishes the assistant role before content deltas arrive. Stream, request, and trace headers identify the request, while final persistence may complete after the client has already observed output.

The supplied handler does not document a resumable stream token or replay cursor. If the client closes the connection, preserve the request identifiers and treat any later retry as a separate request unless the application has its own duplicate-control design. Do not merge partial text from two requests without an explicit product rule.

Errors

When partial text has already reached the user, the application should mark the result incomplete rather than silently replacing the failure with a success state. Whether partial content is retained, hidden, or labeled is a client policy, but the provider-stream error and trace identifiers must remain available.

Treat provider_stream_error as a failed request even if some content has already been rendered. Do not automatically resubmit the request without considering duplicate user-visible output and provider cost. The chat route does not provide automatic idempotency for a repeated streaming request.

Authentication, scope, setup, model-readiness, credential, and budget errors normally occur before SSE delivery begins. If no event stream is established, handle the returned HTTP error using the same classification as a non-streaming request.

A provider-stream failure can emit an error chunk and then [DONE].

Best-effort persistence means a secondary logging or usage write should not be assumed to be transactionally identical to stream delivery.

Inspect the error payload, request ID, trace ID, and attempt records before deciding whether a new request is safe.

A stream can fail after it begins. An error chunk followed by [DONE] indicates a terminated failure path, not a successful completion with an unusual final message.

Before expanding the integration, parse complete SSE events, accumulate text deltas, capture the final usage data, and treat an error chunk as failure even when [DONE] follows.

Stream persistence is best-effort, and the source does not promise reconnect, resume, or cancellation semantics beyond the current HTTP connection.

Differentiate a protocol error from a provider error. Malformed SSE handling can lose boundaries or final usage even when the server behaved correctly, while an error event produced by the Gateway represents a failed request. Preserve the request and trace identifiers in either case, because transport symptoms alone do not identify whether the fault occurred in parsing, routing, or provider execution.