Gateway architecture

A Gateway request crosses several independently testable boundaries: authentication, catalog readiness, policy, routing, provider execution, and observability. This page maps those boundaries so developers and operators can locate a failure without treating the Gateway as one opaque call.

The architecture is implemented, but its surrounding product surface remains Beta or setup-required in some environments.

For architecture reviews, preserve the difference between control-plane information and execution evidence. A route profile, provider setting, alias, timeout, or budget describes how the system intends to behave. The response metadata, attempt rows, request log, and usage event describe the path actually taken. Comparing those two views is more useful than treating the Gateway as an opaque proxy. It also prevents an operator from changing provider settings when the request was rejected earlier by authentication, catalog readiness, policy, or budget enforcement.

Architecture overview

The boundary between Ethen and the external provider begins at adapter invocation. Everything before that point—authentication, validation, budget, routing, policy, credential lookup, aliasing, and timeout selection—belongs to the Gateway’s control plane.

The inbound chat handler owns request validation, project-key verification, runtime-status checks, and budget enforcement.

The routing runtime owns route-profile resolution, provider filtering, policy preflight, retries, fallbacks, BYOK lookup, aliases, and timeouts.

Provider adapters execute the selected provider request, while platform modules record metadata, attempts, and usage.

The architecture separates policy and platform checks from provider execution. That separation is useful during incident review: a request rejected by the handler has no provider attempt, while a routed request can produce one or more attempt records before a final response.

The routing runtime resolves a route profile, removes unavailable or circuit-open providers, and applies request and project constraints. BYOK resolution, model aliases, and provider timeouts are applied immediately before adapter execution.

Policy preflight can exclude a provider before its credential or adapter is used. Each provider attempt records its provider, model, success state, latency, and redacted failure details.

Request lifecycle

After body validation, the handler resolves the authenticated project and checks the requested model against catalog state. Budget enforcement occurs before provider work. Routing then assembles eligible candidates, applies request restrictions and project policy, resolves any configured aliases, and invokes adapters in order. Attempt, request-log, and usage records are written around that execution path so a response can be traced without treating the provider call as the only meaningful stage.

The handler processes a chat request in a fixed order. It first parses JSON and checks for a model and at least one message. It then extracts the Bearer key, verifies the stored hash, rejects revoked or expired keys, and requires the gateway:invoke scope. Model runtime status and project budget are evaluated before provider selection. A request that fails at any of those stages never reaches an outbound provider adapter.

Once the request is eligible, the runtime resolves a route profile and builds the provider order. It removes providers that are unavailable or protected by an open circuit breaker, applies the request-level only restriction, applies the project allow-list, and runs policy preflight. The remaining route candidates are then processed according to the configured retry and fallback policy. BYOK resolution, model alias selection, and provider timeout selection occur for each attempt immediately before adapter execution.

The handler rejects malformed JSON, missing model, missing messages, invalid key states, missing invoke scope, non-runnable models, and exhausted budget before provider execution.

A successful request receives identifiers that connect the response, request log, provider attempts, and usage event.

Streaming and non-streaming requests share the routing path but differ in response delivery and persistence timing.

Request and trace identifiers connect the external response to internal records. Keep them together when comparing logs, attempts, and usage so that retries or fallbacks are not mistaken for independent user requests.

Request logs and usage events are written by platform modules after the runtime returns or streaming persistence completes.

The handler validates JSON, model, messages, key state, scope, model runtime status, and project budget before provider selection.

Routing

Route construction and provider execution are separate stages. Route construction decides which providers remain eligible and in what order they may be attempted. Execution begins only after the runtime resolves the provider credential, provider-specific model alias, and timeout for one candidate. This distinction explains why a policy denial or open circuit can remove a provider without producing an adapter attempt, while a timeout produces an attempt row with latency and failure details.

Provider order can originate from a route profile and be narrowed by request-level only controls and project allow-lists.

Providers are removed when unavailable or protected by an open circuit breaker.

Policy preflight can exclude a route before credentials or an adapter are used.

Routing begins with a profile and becomes narrower as availability, circuit state, request controls, project allow-lists, and policy are applied. Provider order therefore expresses preference, not a promise that the first provider will be selected.

Provider order expresses preference, while eligibility, policy, credentials, circuit state, and timeout settings decide which attempts can occur.

Provider execution

The runtime resolves the provider credential, model alias, and timeout immediately before adapter execution.

An attempt records provider, model, success, latency, and a redacted failure when execution does not succeed.

Same-provider retry and cross-provider fallback counts come from configured policy rather than a universal constant.

Credentials, aliases, and timeouts are resolved for each provider attempt. A provider-specific model alias can differ from the requested model string, and a timeout is recorded as a provider failure rather than a successful but slow response.

A retry adds another attempt against the same provider; a fallback changes the provider route and should be reported separately.

Attempt records remain relevant after a later success because the final provider field does not explain the entire route history.

Observability

The chat handler writes three related records. The request log identifies the request, trace, route, model, final provider, status, latency, token totals, estimated cost, fallback state, error code, project, API key, and user. Provider-attempt rows preserve the sequence of provider and model choices, success state, latency, and redacted failure details. Usage events record request count, input tokens, output tokens, estimated cost, and grouping dimensions such as project, model, provider, route, and trace.

These records answer different questions. The request log summarizes the final outcome. Attempt rows explain how routing reached that outcome. Usage events support aggregation. A later provider success does not erase an earlier failed attempt, and the final provider field alone is not enough to reconstruct a fallback path.

Request logs carry request and trace identifiers, route, model, provider, status, latency, estimated cost, token counts, fallback state, and error code.

Usage records aggregate requests, input tokens, output tokens, and estimated cost by project, model, and provider.

The inspected chat route uses metadata-only logging and does not establish that every other surface follows the same rule.

Observability spans several records rather than one universal event. Request logs describe the overall call, attempt rows describe provider execution, and usage events carry token and estimated-cost information for aggregation.

Redacted error details and identifiers are safer diagnostic material than raw prompts, credentials, or provider responses.

Trace failures in processing order so an authentication or catalog rejection is not misdiagnosed as a provider outage.

The final provider field is insufficient when retries or fallbacks occurred, so the attempt sequence must remain visible.

Failure handling

Validation, authentication, model-readiness, budget, routing, provider, and streaming failures occur at different stages. Request-level errors return before a provider attempt exists. Policy exclusion or route exhaustion may leave no executable candidate. A provider timeout is recorded as a provider failure and can participate in configured retry or fallback behavior. Streaming can fail after a provider has already been selected; the handler then emits an error chunk and a terminal [DONE] marker.

Because retry counts and provider timeouts come from configuration, the documentation cannot define one global value. Diagnose failures from the earliest recorded stage and preserve the request ID, trace ID, model-status header, route metadata, and attempt sequence.

Validation failures stop before routing, while provider failures can create attempts and trigger configured retry or fallback behavior.

Timeouts are treated as provider failures and circuit-breaker state can affect future eligibility.

When no eligible route succeeds, the runtime returns the last relevant error instead of fabricating a successful response.

Failure handling should be read from the earliest failed layer. Validation errors stop before routing; route exhaustion or policy can prevent adapter execution; provider failures create attempts and may lead to retry or fallback; a failed stream can close cleanly after an error chunk.

Streaming and non-streaming share the routing runtime, but delivery and persistence timing differ; provider timeout and retry counts remain configuration-dependent.

The position of a failure determines which records exist. Authentication, request validation, model-status checks, budget enforcement, and policy denial can stop processing before a provider attempt is created. Once an adapter is called, the attempt record can identify the provider, mapped model, latency, success state, and redacted error. This distinction prevents an empty attempt list from being misread as a provider outage.

Request and trace identifiers connect the response to the metadata written by the inspected handler. Usage and log records may then show token counts, estimated cost, route selection, fallback state, and error code. The current source verifies metadata-only logging for this chat path; it does not establish the same storage behavior for every future Gateway endpoint. Architecture diagrams and operational guidance should keep that scope visible rather than turning one handler’s implementation into a platform-wide retention claim.