Providers and routing

A requested model does not by itself determine which provider executes a Gateway call. The routing runtime builds an eligible provider sequence, applies project and request constraints, and records the route that actually ran.

Provider IDs and availability are implementation observations, not permanent commitments.

Routing configuration is best treated as a set of constraints applied to a changing candidate pool. Availability and circuit state remove providers first; request-level controls, project allow-lists, and policy can narrow the pool further; credentials, aliases, and timeouts are then resolved for attempts. This explains why the selected provider can differ from a preferred order without indicating a defect. The response and attempt metadata should be the source of truth for which provider and model alias were actually used.

Provider selection

The mock provider belongs to the runtime contract but should not be confused with a production external route. Any result produced through mock mode must remain labeled so tests do not become evidence of live provider availability.

Provider selection starts with a route profile and the current runtime provider set. The inspected types recognize mock, openai, anthropic, deepseek, and openai-compatible. That list describes the current code contract, not a permanent commercial promise or proof that each provider is configured in every project.

The runtime builds an ordered candidate list, removes unavailable providers and providers held open by a circuit breaker, then applies request and project restrictions. A provider becomes an attempt only after it survives those checks and policy preflight.

Current runtime provider identifiers include openai, anthropic, deepseek, openai-compatible, and mock.

The runtime starts from a route profile or production provider set and removes unavailable or circuit-open entries.

The first successful eligible attempt becomes the selected provider.

Provider selection starts with the route profile and is narrowed by runtime conditions. The requested model does not name the final provider, and the first provider in an order list can be removed before any credential lookup occurs.

Unavailable providers and providers protected by an open circuit breaker are removed. Provider-specific aliases and timeouts are resolved for each attempt.

Aliases translate the requested model for a selected provider and should not be confused with the client-visible model choice. Policy preflight runs before credentials and the provider adapter are used. The runtime currently recognizes mock, openai, anthropic, deepseek, and openai-compatible provider IDs.

Route profiles contribute an initial provider order.

Routing rules

Request restrictions should be narrow and temporary when used for diagnosis. An only list can prove whether a particular eligible provider works, but leaving that restriction in production can remove resilience and make a later provider outage appear to be total Gateway failure.

Request-level options can specify provider order and an only set. Project settings can contribute an allow-list. The intersection of those controls narrows the candidate route; it cannot make a provider available, add missing credentials, override a policy denial, or repair an unsupported model mapping.

Provider-specific model aliases translate the requested model into the identifier required by a selected adapter. Per-provider timeouts are also resolved at the attempt level. Keep the original requested model, selected provider, and selected alias as separate fields when interpreting a response.

Request controls can set provider order, an only allow-list, provider-specific model aliases, and provider timeouts.

Project provider allow-lists can narrow the request further.

Configured fallback policy determines retry and cross-provider behavior after the initial route is built.

Routing rules can express preferred order, an only set, provider-specific model aliases, and timeouts. Project allow-lists and policy preflight further restrict the candidates; they are not optional client hints.

Route profiles contribute an initial provider order.

Request-level only controls and project allow-lists can narrow that order. Provider order expresses preference, while eligibility, policy, credentials, circuit state, and timeout settings decide which attempts can occur.

Availability

A provider removed by the circuit breaker should remain distinguishable from one excluded by request or project policy. Both can result in no attempt, but they imply different operational ownership and recovery. Circuit state belongs to runtime health, while allow-lists and only restrictions belong to configuration.

Provider health and circuit state are evaluated alongside static configuration. A configured provider can still be excluded temporarily, and a provider listed in the runtime types can remain unavailable to the project. Operational dashboards should therefore display current eligibility separately from configured presence.

A candidate can be removed because the provider is unavailable, its circuit breaker is open, the project disallows it, request options exclude it, policy preflight rejects it, credentials are missing, or the model cannot be mapped. Catalog presence alone is not enough.

Availability is evaluated at request time. A provider that executed an earlier request may be ineligible later because configuration, credentials, policy, health, or circuit state changed. Documentation and application logic should therefore treat route metadata as an observation about one request.

Catalog presence does not establish provider configuration or runnable status.

A route also depends on credentials, circuit-breaker state, adapter availability, and model support at the provider.

Runtime availability should be checked per request rather than copied into evergreen documentation.

Availability includes more than a configured adapter. Provider health, circuit-breaker state, project permission, credential presence, catalog readiness, and modality support can all make a route ineligible.

Policy constraints

Eligibility is reduced in stages. Request-level only restrictions can narrow the provider set, project allow-lists can exclude otherwise configured adapters, policy preflight can deny a candidate, and a circuit breaker can temporarily hold an unhealthy route open. BYOK requirements and provider credentials are evaluated separately from the inbound Gateway key. A provider reaches execution only after all applicable gates pass.

Warnings should be retained when the runtime narrows or changes a route without failing the request. They can explain why the selected provider differs from the preferred order even when the final response is successful.

Policy preflight runs before the provider credential is resolved and before the adapter executes. A denied provider should not receive the outbound request or its credential through the normal route. Project allow-lists and request restrictions are also applied before adapter execution.

Routing options are not an escape hatch around policy. If every candidate is removed, the correct result is a routing or readiness failure, not an attempt against a disallowed provider.

Policy preflight runs before provider execution and can remove otherwise configured routes.

A request-level preference cannot override a project prohibition or a blocked policy result.

BYOK resolution occurs only after a provider remains eligible.

Policy constraints are applied before provider execution. This gives the Gateway a place to reject a disallowed route without sending data or invoking an external adapter.

Observability

Each attempt identifies the provider, provider-specific model, success state, latency, and redacted error details. Final metadata identifies the selected provider, selected alias, attempt count, fallback state and reason, routing mode or source, and warnings.

Read attempts in sequence. Provider order describes preference, while the attempt list shows what actually happened. A final success after two failures should be represented as three attempts with fallback metadata, not as a single uncomplicated provider call.

Route metadata records the selected provider, selected model alias, attempts, fallback state, source, mode, and warnings.

Provider-attempt records add success, latency, and redacted error details.

Use request and trace IDs to connect response metadata to logs.

Observability should retain the route source, selected provider and alias, attempt sequence, warnings, fallback status, and reason. Those fields explain why the runtime diverged from a request preference.

Route profiles contribute the initial provider order. Request-level only controls and project allow-lists can narrow it, while an empty inspection result should not be interpreted as deletion or expiration without a verified retention rule.

Read request and trace identifiers first, then correlate route, provider attempts, latency, errors, and usage in processing order.

No provider attempts usually points to an earlier validation or policy gate; populated attempts shift attention to routing or execution.

Metadata-only logging is verified only for the inspected chat handler and should not be generalized to every route.

Warnings deserve attention even when the completion succeeds. They can reveal that the preferred provider was skipped, an alias changed, or fallback altered the route. Recording those warnings with the attempt sequence gives operators a way to detect degraded routing before it becomes a complete outage.

Examples

A second case illustrates aliases. The client requests model M, the route selects Provider B, and the project mapping resolves M to provider alias B-M. The response should keep the original requested model in application context and report B-M as selected route metadata. Replacing the client’s model value with the alias would make later requests provider-specific and defeat the routing abstraction.

A timeout case is different from preflight removal. If Provider A is removed because policy denies it, there is no adapter timeout to inspect. If A remains eligible and exceeds its configured timeout, the router records a failed attempt and can continue according to fallback policy. That difference should remain visible in logs and operational dashboards.

Suppose the route profile prefers Provider A, then Provider B. If A is unavailable before execution, the runtime may remove it without creating a provider call and continue with B. If A is eligible but its adapter times out, the timeout becomes a failed attempt; configured retry or fallback policy then determines whether A is tried again or B is attempted.

If the request supplies only: [B], A should not be used even when the profile prefers it. If project policy rejects B, no candidate remains. This example illustrates restriction and eligibility; it does not define a global retry count or guarantee that the named providers exist in a particular deployment.

An order list expresses preference but does not guarantee that its first provider will run.

An only list restricts eligibility and can leave no usable provider when credentials or policy remove the remaining entries.

A models mapping changes the provider-specific alias without changing the requested model identifier.

Examples should use provider IDs and controls that appear in the source bundle, while making clear that the IDs describe the current runtime rather than a permanent public support matrix.

Provider identifiers in the runtime are implementation facts, not permanent availability guarantees; credentials, policy, allow-lists, health, and circuit state can remove a provider.

Consider a route profile that prefers Provider A and then Provider B. If A is removed because its circuit is open, no A adapter call is made; B can become the first attempt. If A remains eligible but exceeds its provider-specific timeout, the attempt history records the failure and the configured fallback policy determines whether B is tried. A policy denial is different again: the candidate is rejected before its credential or adapter is used.

Aliases add another layer without changing the client’s requested model. A project can map the requested identifier to an adapter-specific model for the selected provider. The response should expose that selected alias in route metadata while the application keeps the original model choice. This separation allows the same client request to follow a different eligible provider later. Provider identifiers in the runtime types describe the current implementation, not a guarantee that every project has those providers configured or that the list will remain permanent.