Fallbacks and retry behavior

Retries and fallbacks are related recovery actions with different operational consequences. A retry repeats execution against a provider under configured policy, while a fallback moves to another eligible provider or route.

No universal retry count, timeout, or chat-request idempotency guarantee is verified.

Resilience settings should be tuned against observed failure classes rather than a desired attempt count. Transient provider failures may justify another attempt, while missing credentials, policy restrictions, catalog-only status, or budget enforcement require a configuration change. Attempt records make that distinction visible. They also show the latency and cost tradeoff of retrying before falling back. Because the request contract does not guarantee idempotency, clients should consider duplicate work when resubmitting after uncertain transport failures.

Failure classes

Classifying a timeout also requires knowing which timeout fired. A client deadline, Gateway provider timeout, and external provider failure can produce similar user symptoms while creating different records. Preserve timing and attempt information before deciding that the provider should be retried.

Not every failure is eligible for retry or fallback. Invalid JSON, missing model or messages, key errors, insufficient scope, setup-required states, catalog-only models, budget enforcement, and policy exclusion occur before a provider adapter can succeed. Repeating the provider route does not correct those conditions.

Provider-stage failures include adapter errors and timeouts. Streaming can also fail after delivery starts. Those failures create provider-attempt evidence and may be processed by the configured fallback policy.

Validation, authentication, model readiness, policy, budget, credential, timeout, adapter, and stream failures occur at different stages.

Failures before routing generally do not create provider attempts.

Provider execution failures can update circuit state and become candidates for retry or fallback.

Failure classes determine whether another attempt is meaningful. Validation, authentication, catalog, policy, and budget failures occur before a provider call; repeating them against another provider does not repair the underlying cause.

When no eligible attempt succeeds, the runtime throws the last relevant error. Same-provider retries are resolved separately from cross-provider fallbacks.

The chat request contract does not establish automatic idempotency for repeated client submissions. A provider timeout records a failed attempt and can affect circuit-breaker state.

Each attempt identifies provider, model, success, latency, and a redacted error when it fails.

Retry behavior

Attempt history is the authoritative view of resilience behavior for a request. Each failed adapter call can record its provider, timing, and error before policy decides whether another attempt is allowed. Because retry counts are configuration-dependent, client code should not assume a fixed number or infer that every transient failure will be repeated. Repeated authentication or policy failures usually require correction rather than another attempt.

A successful retry on the same provider should still retain the earlier failed attempt. Otherwise latency and reliability analysis will undercount transient failures and make the provider appear healthier than the request history supports.

The runtime’s retry count is taken from configured policy rather than a fixed global constant. Each attempt should remain visible with its provider, model alias, latency, success state, and redacted failure details.

Retries are appropriate only when the route still considers the provider eligible. An open circuit breaker, policy denial, missing credential, or unsupported mapping removes the provider rather than making repeated adapter calls useful.

Same-provider retry counts come from the configured fallback policy.

Each execution attempt records its provider, model alias, success, latency, and redacted error information.

Repeating an invalid, revoked, expired, under-scoped, or budget-blocked request does not fix the underlying gate.

A retry repeats execution against the same provider according to configured fallback policy. It should create a separate attempt record so latency and redacted failure information remain visible.

A fallback can use a provider-specific model alias rather than the original model string.

A provider timeout produces a failed attempt and can affect circuit-breaker state. When every eligible attempt fails, the runtime returns the last relevant error.

Fallbacks

Fallback metadata should remain visible to callers that care which provider executed.

A fallback changes the provider route. The replacement provider may use a different model alias and timeout. Final metadata reports whether fallback was used and can include a fallback provider and reason, while the attempt list preserves the complete route history.

Fallback does not mean every provider is interchangeable. Project allow-lists, request only restrictions, policy, credentials, availability, and model support are applied to each candidate. If no eligible attempt succeeds, the runtime returns the last relevant error.

Cross-provider fallback uses the remaining eligible provider order after project, policy, availability, and circuit filters.

The response metadata can identify whether fallback occurred and why.

Fallback can change latency, provider behavior, and estimated cost, so it should remain observable.

A fallback changes the provider and can also change the provider-specific model alias. The response metadata should identify that change instead of presenting the final provider as though it were the original route.

Timeout failures are recorded, but the chat request contract does not establish automatic idempotency for repeated client submissions.

Aliases translate the requested model for a selected provider and should not be confused with the client-visible model choice.

Idempotency

The chat-completions handler does not provide automatic request idempotency in the supplied contract. A caller that resubmits after a timeout may create another provider operation and another usage event, even when the first result was merely delayed or the stream was partially delivered.

Applications that need duplicate protection must design it outside the documented chat request shape. Do not invent an Idempotency-Key header or request field for this route. For non-idempotent user actions, confirm the original request state through logs before resubmitting.

The inspected chat request type does not establish an idempotency key or automatic de-duplication contract.

A client retry may therefore create a separate provider request and separate usage record.

Callers should apply their own operation-level safety where duplicate generation would be harmful.

A client that resubmits after timeout or disconnection should consider whether the provider may already have processed the original request.

Same-provider retries remain separate from cross-provider fallback. Model aliases translate the client-visible identifier for the selected provider, and a timed-out attempt can update circuit-breaker state.

Retry policy inside routing does not create a second client request; application retries do. If the caller resends the chat request after an uncertain network result, the inspected contract offers no idempotency field that proves duplicate suppression. Design callers so repeated side effects outside the model response are controlled by the application.

Timeouts

A timeout value should be reviewed with the provider and workload it applies to. The same duration can be too short for one model and unnecessarily long for another, so documentation should not promote one default.

Provider timeouts can be configured per provider through Gateway routing options. A timeout is treated as provider failure and may feed retry or fallback behavior. Timeouts are configuration-specific rather than one value shared by every provider, model, route, and project.

Client timeouts and provider timeouts should be recorded separately. A client may stop waiting before the provider timeout fires, while the Gateway can still be processing the attempt. Use request and trace identifiers to determine whether the runtime completed later.

Per-provider timeout controls can be supplied through gateway provider options.

A provider timeout is treated as a provider failure and may influence circuit and fallback behavior.

Provider and deployment timeout values remain environment-specific.

Provider timeouts are configuration-dependent and are treated as provider failures. Do not publish one timeout value for the entire Gateway or assume that every provider shares the same threshold.

A provider-specific timeout limits one eligible attempt. It does not define a workflow-wide deadline or guarantee that the next provider will have the same limit.

Operational guidance

If attempts fail across several providers, compare their error classes before changing policy. A shared credential, model-readiness, or project restriction can affect every candidate; increasing the provider list will not repair that common cause. Conversely, one adapter timeout with later fallback success points to route health rather than request validation.

Use automatic resilience only for failure classes that are safe to repeat. A timeout may be transient, but the caller still needs to consider whether the provider could have generated billable work before the timeout was observed. Streaming raises an additional duplicate-output risk because the user may already have seen partial text.

When evaluating a fallback route, compare quality and operational behavior separately. The fallback may use a different alias, provider account, latency profile, or estimated cost. Record the fallback reason and warning fields so the application can distinguish a normal primary route from degraded operation.

Begin with a route that has no request-level overrides, then introduce provider order, restrictions, retries, and fallback one change at a time. Inspect the attempt sequence after each test. This makes a policy or credential exclusion distinguishable from an adapter failure.

For production handling, classify errors before retrying. Never retry authentication or budget failures as provider outages. Limit automatic retries around operations where duplicate output or cost would be harmful, and retain fallback metadata for later analysis.

Classify the failure before retrying and preserve the original request and trace identifiers.

Review attempt history to determine whether the route changed or repeatedly failed at the same provider.

Restrict providers deliberately when reproducibility matters, and allow fallback only when the resulting variability is acceptable.

Operational tuning should start with attempt evidence. More retries can increase latency and cost without improving success when the cause is a missing key, policy denial, model status, or persistent provider outage.

Retry and fallback counts come from configured policy, no global count is verified, and provider timeouts are treated as provider failures.

Read the attempt sequence before deciding whether resilience worked. Multiple attempts against the same provider indicate retry behavior; a change in provider identifies fallback. The final metadata can report whether fallback occurred and why, while each attempt contributes its provider, mapped model, latency, success state, and redacted error. When no attempt exists, investigate validation, readiness, policy, budget, or route construction instead of increasing retry settings.

Timeouts are provider failures only after an eligible adapter is called. A project or request restriction that removes a provider is not a timeout and should not consume a retry. Likewise, the chat contract does not define automatic request idempotency, so repeating a failed request can produce another provider invocation. Applications that need duplicate protection must design it explicitly rather than assuming the Gateway will collapse retried chat calls.