Understanding pricing, latency, speed, and context
Interpret model pricing, latency, throughput, context windows, and output limits without freezing dynamic or missing values.
Understanding pricing, latency, speed, and context
Pricing, latency, throughput, context, and output limits describe different operational constraints. Use this page when two models appear similar on quality but differ in cost or performance, or when a request may exceed the current model record. These values can be dynamic or missing; documentation should teach interpretation rather than freeze numbers.
Pricing units
The gateway catalog can represent input price, output price, cache read price, cache write price, and web-search price. A value is meaningful only with its unit, direction, provider, and current source.
| Pricing field | What it generally represents | Interpretation boundary |
|---|---|---|
| Input price | Cost associated with supplied context | Do not combine with output price without workload estimates |
| Output price | Cost associated with generated output | Long responses can change the cost balance |
| Cache read price | Cost for supported cached-input reuse | Presence does not prove caching is enabled for the route |
| Cache write price | Cost to create supported cached state | Must be tied to actual provider behavior |
| Web-search price | Cost associated with supported search behavior | Do not assume every model or product invokes search |
The source pack does not define a universal billing formula or currency presentation. Use the current product fields. Missing price is unknown, not free.
Estimate cost from representative task shape and verify the result against the provider-linked record. Include retries and human review when comparing total operating cost.
The gateway catalog can store input, output, cache-read, cache-write, and web-search price fields. The current type definition does not guarantee that every field shares one unit or is populated for every model.
Read the visible label and current provider evidence before calculating cost. Missing data is unavailable, not free. Do not copy a current price into evergreen documentation.
Latency
Latency describes response time, but one number can represent different measurements. The normalized adapter groups latency and response-time charts separately from speed and throughput, indicating that these concepts should not be merged.
Latency can be affected by:
- provider route;
- model and version;
- prompt and output length;
- modality;
- network conditions;
- service load;
- tool or search calls;
- local runtime and hardware for local paths.
Use represented latency as directional evidence. Test the intended route with realistic requests before making a user-experience decision. Do not call a model “fastest” based on a missing or incomparable field.
For interactive work, consistency may matter as much as the average. Batch 01 does not provide percentile or measurement-method guarantees, so avoid statistical claims not shown by the source.
Latency represents waiting time associated with beginning or completing a response, depending on the metric label. Model Intelligence distinguishes latency and response-time chart groups.
Check the exact metric before comparing. Provider load, reasoning behavior, input size, and environment can change observed latency.
End-to-end time
Model latency is only one component of user-perceived time. Context preparation, routing, provider access, tools, search, output length, and review can all contribute. Measure the path the user actually experiences rather than attributing every delay to the model.
For a tool-assisted task, preserve separate observations when possible: time to model response, time spent in the tool or integration, and time to target verification. This makes optimization decisions more precise without inventing unsupported telemetry fields.
Cost of uncertainty
When a key field is missing, the cost of obtaining evidence may be lower than the cost of selecting blindly. A small controlled test can reveal whether latency or output behavior fits the workload. For dynamic price or limit questions, use the current provider-linked source rather than extrapolating from another model.
Throughput
Throughput is the rate at which work is processed. It is relevant to batch tasks, long outputs, or repeated requests and should not be confused with time to first response.
A route can have acceptable latency for one request while offering limited throughput under volume. Another route can process substantial work efficiently while feeling slower interactively.
Compare throughput only when units and conditions match. Consider:
- request concurrency;
- input and output length;
- model/provider route;
- modality;
- local device resources;
- tool or integration overhead;
- rate or quota constraints, if verified elsewhere.
The current sources represent a throughput field but do not define a universal test method. Missing throughput remains unavailable.
Throughput represents generation speed, commonly shown as output speed in the current presentation groups.
Throughput should be read with latency and total task duration. A model can start slowly and generate quickly, or start quickly and produce slowly. The best tradeoff depends on the workload.
Context windows
A context window is the model record’s represented capacity for request context. It is not the same as workspace context, session history, or total stored material.
Use context capacity in four steps:
- Estimate how much source material must be processed together.
- Reserve space for instructions and expected output.
- Confirm the current context field for the exact model/provider record.
- Test retrieval and reasoning quality with representative long input.
A long-context capability family is a discovery signal. It does not replace the numeric field, and a large field does not guarantee equal quality throughout the full range.
Context can affect price and latency because larger inputs require more processing. Splitting work into reviewable stages may be better than maximizing capacity.
Context window is a verified catalog field and Model Intelligence comparison dimension. It describes available input/context capacity when supplied.
Larger context can support longer material, but it can also affect cost and latency. The field may be absent, and supported practical context may depend on the route and provider.
Output limits
Maximum output tokens are represented separately from context windows. A model can accept a large input and still have a smaller output limit.
Before requesting a long artifact, check whether the output can be divided into sections or generated iteratively. Preserve the structure and review points so multiple outputs do not lose context or introduce contradictions.
The current output field may be missing or dynamic. Do not infer a limit from another model with a similar name, and do not convert absence to unlimited capacity.
If a response ends early, distinguish among an output limit, product behavior, prompt design, error, or user interruption. The Batch 01 sources do not define exact truncation messages.
Maximum output tokens is a separate verified catalog field. It can be missing and should not be inferred from context window size.
Before relying on a long output, verify the current model, route, and provider limit. The current product surfaces do not define truncation behavior or defaults when the field is absent.
Cost-performance tradeoffs
A model decision should combine the fields rather than optimize one in isolation.
| Workload | Primary concern | Secondary checks |
|---|---|---|
| Interactive assistant | Latency and quality | Output length, provider status, cost |
| Large-document analysis | Context fit and quality | Input price, latency, evidence coverage |
| Batch transformation | Throughput and cost | Output limit, retry rate, review effort |
| Long-form generation | Output capacity and quality | Output price, latency, consistency |
| Tool-assisted workflow | Reliability and verification | Model latency, tool latency, approval overhead |
| Local execution | Device performance and operational fit | Context, model support, runtime evidence |
Use a current comparison record rather than an evergreen numeric recommendation. Capture model ID, provider, date, visible status, populated fields, and representative test results.
When one field is missing, do not force a ranking. State the uncertainty and decide whether another candidate, a direct test, or a later verification step is necessary. Pricing and limits retain explicit review flags because the fields exist but their current values and units require ongoing confirmation.
Evaluate cost, latency, throughput, context, output limits, quality evidence, and runtime availability together.
A cheap input price can be offset by long outputs; high throughput can be offset by initial latency; a large context window can increase cost without improving a short task. Use representative requests and current data rather than a permanent ranking.
Estimating from workload shape
Begin with representative input and output sizes rather than a hypothetical maximum. Identify how often the task runs, whether context is reused, whether search or tools are involved, and how many retries a normal review requires. Then apply only the pricing fields and units currently shown for the exact provider record.
This estimate remains provisional because Batch 01 does not define billing aggregation, currency, rounding, or provider-specific charging rules. Use it to compare candidates, not as an invoice prediction.
Missing-value decisions
A missing price, latency, throughput, context, or output field creates an information gap. Choose among three responses: obtain the current value from an authoritative product source, test the operational behavior directly, or remove the candidate when the unknown is too important.
Do not rank missing data as best, worst, or average. Record “unavailable” in the comparison and explain how the decision handled it.
Interactive versus batch profiles
Interactive tasks care about the time a user waits and the consistency of the response. Batch tasks care about total throughput, cost, and failure recovery. A model can be suitable for one and unsuitable for the other even when quality is similar.
Separate the test sets. Use short, realistic interactions for the interactive path and representative volumes or document sizes for the batch path. Do not infer concurrency or service limits that the current sources do not provide.
Context budgeting
Reserve part of the represented context for instructions and output. The entire window should not be treated as usable source capacity. Remove duplicated material, split unrelated documents, and summarize only when the summary remains traceable to the source.
When a task must span multiple requests, preserve the accepted decisions and source references between stages. A workspace can organize the material, but the request context still needs deliberate assembly.
Output planning
Long outputs can increase price, latency, review effort, and the chance of inconsistency. Define the required structure and generate only the sections needed. If the output limit is unknown, use smaller reviewable segments rather than assuming the model can complete one large artifact.
A continuation should restate the relevant constraints and identify the prior accepted section. It should not rely on unverified hidden memory.
Recording observed performance
For a test, capture model ID, provider, route, date, input shape, requested output, visible status, represented catalog fields, and observed result. Avoid presenting one observation as a universal latency or throughput benchmark.
Observed performance can still improve selection. It is direct evidence for the tested workload and environment, provided the record makes its scope clear.