Observability, traces, errors, and evaluations

Ethen’s observability surfaces summarize runs, receipts, latency, tokens, cost estimates, errors, tool-call outcomes, and evaluation-related views where data exists. The query layer also labels data honesty and can fall back to fixtures.

This page explains how to interpret bridged, fixture, live, and unavailable information. Complete tracing, trusted cost attribution, TTFT, fallback-rate coverage, and a durable evaluation-governance lifecycle are not established.

Observability model

Observability begins with emitted records. A dashboard can summarize only the runs, receipts, traces, errors, tokens, and costs supplied by its source runtimes.

lib/platform/observability/queries.ts reads platform runs and receipts, computes completion and error rates, aggregates latency, tokens, and costs when present, and summarizes tool-call success when receipts exist. It labels data honesty and warns when fixtures are involved.

Do not interpret a populated dashboard as complete coverage. Some runtimes do not expose model-call receipts or traces.

Data-honesty states

The observability layer should preserve four meanings:

• live: emitted by the current runtime for real activity;
• bridged: read from another supported store or compatibility layer;
• fixture: generated demonstration data;
• unavailable: the metric or record is not emitted.

These labels are part of the metric, not decoration. A fixture completion rate must not be compared with live provider errors. An unavailable TTFT should remain absent rather than zero. A bridged token count should identify the store that supplied it.

The query can summarize only the records it receives. If a runtime produces a final output without a model-call receipt, the dashboard can show completion while lacking model latency, tokens, cost, or provider-attempt details.

Coverage inventory

List each production runtime and the records it emits: run, receipt, trace, model call, provider attempt, token usage, cost estimate, tool result, and error. Mark absent fields as unavailable rather than assuming another runtime’s schema applies.

This inventory explains why a dashboard can be complete for one product and sparse for another. It also gives engineering a concrete backlog for emitters and correlation.

Trust levels for metrics

A metric can be suitable for debugging without being suitable for billing or an executive report. Define the decision the metric supports and the completeness required for that decision.

For example, a fixture latency chart can verify layout, a bridged provider error can support diagnosis, and a complete live usage ledger would be needed for financial reconciliation. Do not promote one trust level into another.

Traces

A trace connects related operations through identifiers and parent-child relationships. The /observability/traces route provides a trace-oriented view, but not every request, provider attempt, tool call, or workflow emits a complete trace.

When a trace exists, use it to follow ordering and latency across components. Compare it with request IDs, run IDs, and provider identifiers rather than treating those fields as interchangeable.

TTFT is currently unavailable, and fallback-rate metrics are also unavailable. Do not calculate them from unrelated fields without a supported definition.

Correlation model

Keep trace ID, request ID, run ID, receipt ID, provider request ID, and project ID distinct. A trace can contain several requests or spans; a run can create several receipts; a provider can return its own identifier for one attempt.

A troubleshooting view should use these relationships to move from the customer-visible action to the runtime and provider boundary. If the relationship is missing, the gap should be reported instead of joining records by time alone.

Trace coverage should be tested per runtime. Gateway, agents, Studio, Voice, Sentinel, and local paths can emit different records. The route family does not prove a universal tracing library or complete propagation.

Sampling and missing spans

No universal sampling policy is documented. If a runtime samples traces, the dashboard needs to distinguish sampled absence from emission failure. If it does not sample, high-volume storage and retention become separate operating decisions.

A partial trace can still help when its boundaries are clear. Do not present a root span without provider or tool children as a complete end-to-end trace.

Cross-runtime boundaries

A workflow can call Gateway, a tool, or an external provider. The parent run may exist even when the child runtime emits no compatible trace. Record that break rather than creating a synthetic continuous trace.

If a bridge later connects the records, label the joined data as bridged and retain the original identifiers.

Errors

Error views can group failures by code, route, provider, model, or runtime when those fields are present. A safe error should communicate the actionable boundary without exposing raw secrets or internal implementation details.

Fixture errors demonstrate display behavior but do not establish production incidence. Bridged errors depend on the source system and may omit context.

Troubleshooting should begin at the producing runtime. Determine whether the failure is authentication, policy, provider, limit, storage, execution, or telemetry related before escalating.

Error classification

Classify errors by the component that can correct them:

A safe user-facing error can omit internal details while preserving an error code and correlation identifier. Raw provider bodies and secrets should not be copied into general dashboards.

Error ownership

Assign recurring errors to the team or dependency that can correct them. Authentication and project ownership belong to identity and application operators; provider entitlement belongs to the provider-account owner; telemetry ingestion belongs to the observability path.

A dashboard without ownership becomes a list rather than an operating system. The current product does not prove assignment or remediation workflow, so customers may need an external process.

Recurring error reports should distinguish unique incidents from repeated attempts of the same failing action. Without deduplication or correlation, a retry loop can make one root cause look like many unrelated failures.

Evaluations

An evaluation is a scored assessment of an output or system behavior. The /evals route establishes a product surface, but complete dataset versioning, reviewer assignment, scoring governance, comparison policy, and promotion gates are not verified.

A score should identify its dataset or input set, evaluator, method, model or system version, and time. Without those attributes, comparisons can be misleading.

Evaluation results are evidence for a decision, not a guarantee of quality or safety.

Minimum evaluation record

A reviewable evaluation should identify the input or dataset, system or model version, evaluator type, rubric, score, timestamp, and any reviewer decision. Where human review is used, identify the reviewer role without exposing unnecessary personal information.

The current evaluation surface does not establish dataset version control, blind review, inter-rater policy, automatic promotion, regression gates, or durable reviewer assignment. Those capabilities should remain unclaimed.

Avoid comparing scores created by different rubrics as if they share one scale. A composite result should explain its components and weighting when those details exist.

Avoiding misleading promotion

Do not use one evaluation score as an automatic production gate unless the dataset, threshold, system version, and failure handling are approved. A model can improve on one rubric while regressing on latency, cost, safety, or another task.

The current route supports evaluation concepts, not a complete release-governance system. Keep promotion decisions separate and reviewable.

Review discipline

Use evaluations to compare a defined change, not to certify the platform. Keep baseline and candidate inputs equivalent and record failures as well as aggregate scores.

A result should remain draft or experimental when dataset ownership, scoring method, or reviewer governance is unresolved.

Dashboards

The main observability dashboard and specialized routes for costs, agents, and models can organize operational data by audience. Preserve the honesty label for every metric.

Use bridged when the value comes from another supported store, fixture when it is demonstration data, and unavailable when the runtime does not provide the field. Do not replace unavailable metrics with zeros.

Cost dashboards remain operational estimates rather than billing records.

Reading an aggregate safely

Completion rate requires a known denominator and clear definition of completion. Error rate can vary depending on whether rejected validation, provider attempts, and cancelled work are included. Latency can describe total request time, provider time, or another stage. Cost can be estimated or trusted provider data.

Read the metric definition and honesty state together. If the definition is absent, use the dashboard for orientation and inspect the underlying records before making a governance or financial decision.

A dashboard should not combine fixture and live values into one unlabeled number. The current query warns when fixture data is present; preserve that warning in documentation and reviews.

Dashboards should show the selected time range, project or product scope, and data-honesty state near the metric. Without that context, two viewers can interpret the same number differently. No universal dashboard filter or ownership contract is assumed by this page.

Operators should be able to move from an aggregate to the underlying records where the product supports it. If no drill-down or identifier exists, label the dashboard as summary-only for that metric.

A summary should also show when it was last refreshed so operators can distinguish a quiet system from a stale query result.

Troubleshooting

If a dashboard is empty, check whether the producing runtime is configured, whether it writes runs or receipts, whether the query can reach the backend, and whether the selected time or project scope contains data.

If values look inconsistent, compare the raw record source with the aggregate. Duplicate or missing events, fixture fallback, ownership gaps, and delayed persistence can affect totals.

Keep the relevant request, trace, run, or error identifiers when asking for help. The Help and Contact routes provide assistance without a documented response-time commitment.

Missing or stale telemetry

Check whether the operation actually ran, whether its runtime emits the expected record, whether the backend is configured, whether ownership or time filters exclude it, and whether fixture fallback is active. A page reload cannot repair a runtime that never emitted the trace.

For stale values, compare event time, ingestion time, and dashboard query time where those fields exist. Delayed persistence can make a successful request appear absent temporarily.

When escalating, include the product, route, data-honesty state, identifier, and expected metric. Do not claim an observability SLA or complete trace coverage.

If fixture fallback appears unexpectedly in a production review, stop using the aggregate for decisions and determine why live records are absent. A populated fixture dashboard is not a safe substitute for missing production telemetry.

After restoring telemetry, verify both new records and the expected gap in historical data. A repaired emitter does not reconstruct events that were never written unless another source can supply them.