Skip to content

jaz.llm

New to JAZ? This is generated API reference. Start with the docs and the concepts overview.

Public API of the jaz.llm submodule (import from jaz.llm, e.g. from jaz.llm import ...).

The litellm backend routes through the LiteLLM library and is the default, reaching every provider LiteLLM supports. GitHubCopilotLLM and SGLangLLM ship alongside it for the two cases it does not serve well — completions billed to GitHub AI credits, and a self-hosted SGLang server addressed token-in/token-out. MockLLM answers from a callable you supply, for deterministic tests with no provider. Add your own by subclassing BaseLLM — see BaseLLM for what a backend owns.

Bases: ABC

One LLM backend: the API call, retry, and the cost and model metadata it reports.

Subclass this and pass an instance to add a backend:

class MyLLM(BaseLLM):
def __init__(self, model, base_url, retry_policy=RetryPolicy(), **query_params):
# retry_policy is named above, or the tail would take it
super().__init__(retry_policy=retry_policy)
self._model = model
self._base_url = base_url
self._query_params = query_params # yours; the base never sees them
def complete(self, messages) -> LLMResponse:
wire = ... # call the API with self._model, messages, **self._query_params
return LLMResponse(content=wire.text, prompt_tokens=..., extra={})

LLMResponse is the whole return contract. With this class, and the two value types RetryPolicy and ModelInfo, it is everything a backend author needs from jaz.llm.

Declaring __init__ is all the config wiring a backend needs: its parameters are what authored config can set, and a **query_params tail of its own is how it takes anything it does not name. Give the class a __repr__ that renders that constructor call, with every secret as '***': it is how the console’s settings view and a trace describe a configured backend, shown verbatim and filtered by nothing else — so a dataclass-generated repr would print an api_key field as is. Without a __repr__ they show the class name and model_info.name. The backend stores what it was given and puts it on the request itself; complete receives the messages and nothing else. Nothing passes through this class: a backend that takes a tail names retry_policy explicitly and forwards only that.

To read a key from the shared credentials store (~/.jaz/credentials.json), call jaz.credentials.resolve_credential with the provider’s tag — the default LiteLLM backend does this per request, keyed by the provider it routes the model to. A backend that never calls it does not read the store, so a key stored under its tag is inert. Resolve it below an explicitly passed api_key and below the provider’s environment variable, and hold it as your own attribute so the secret never becomes construction config.

Retry is not a framework layer wrapped around backends: the policy is one construction field of this base class, retry_policy (a RetryPolicy), and the framework runs complete under it. That is what lets the config layer route it with no special knowledge — it is discovered from this __init__ signature exactly as a backend’s base_url is discovered from its own.

What is retried is declared by retryable_exceptions: a failed call is retried only if it raised one of those classes, by default RateLimitError, ServerError and APIConnectionError (which includes APITimeoutError). So raise a transient provider failure as one of them, or add your own class to the tuple; anything else (a rejected request, a bug, a cancellation) propagates at once, unless the RetryPolicy adds its class with extra_retryable_exceptions. When the provider says how long to wait or whether to retry, pass that on the error as retry_after (seconds) and is_retryable: the retry follows the verdict and waits retry_after plus up to a quarter of jitter, except that a retry_after longer than RetryPolicy.retry_after_max means no retry at all, whatever the verdict.

The model is yours too. The base has no model: a backend that serves a nameable model takes model= as its own constructor parameter and reports it through model_info; one that does not (a fixed local server) has nothing to declare.

What you know about the model is reported once, through model_info (a ModelInfo): the name observability hooks label spans with, the input-token limit ContextWindowWarning sizes its window from, and whether you set LLMResponse.cost_usd — the framework prices nothing itself. Every field has an “unknown” default, and each is honoured: a cost_budget on a backend that reports no cost is rejected before any call rather than failing partway through, and an unknown window leaves the context-window hook disabled rather than sized from a guess.

The one part that cannot be construction-time is the retry observer: the core loop passes an on_retry callback carrying invoke_id/iteration to dispatch LLMQueryRetry, which is per-call observability rather than policy.

__init__(*, retry_policy: RetryPolicy | Mapping[str, Any] = RetryPolicy()) -> None
complete(messages: list[Any]) -> LLMResponse

Execute a completion call against the LLM.

The messages are the only input. Everything else the call needs — the model, the sampling parameters, credentials — is the backend’s own state, set at construction and put on the request by this method.

Parameters

  • messages (list[Any]) — The conversation in OpenAI chat format, in wire form — the term used throughout jaz for what actually goes to the provider: the agent’s message buffer with every reserved sidecar key stripped except those this backend lists in extra_message_keys.

Returns

  • LLMResponse — An LLMResponse with the completion content and usage information.
async acomplete(messages: list[Any]) -> LLMResponse

Async version of complete(). Default: runs complete() in a thread.

Subclasses should override this for native async I/O.

retryable_exceptions: tuple[type[BaseException], ...]

Exception types a failed call is retried on by default; anything else is re-raised at once, unless RetryPolicy.extra_retryable_exceptions adds it. The provider can override this for one error (LLMError.is_retryable, from its x-should-retry header).

The default is jaz’s transient classes: RateLimitError, ServerError (a server error or invalid response) and APIConnectionError (a failed transport, timeouts included). Override to add a class your backend raises for a transient failure, for example (*super().retryable_exceptions, httpx.ReadTimeout).

model_info: ModelInfo

What this backend knows about its model — see ModelInfo.

The base knows nothing: no name, no input-token limit, no cost. Override to report what yours can; LiteLLM answers from LiteLLM’s own model map.

Bases: ABC

Renders chat messages to token ids, one message at a time.

A conversation’s generation input is assembled as:

initial_ids + render_message(m_1) + ... + render_message(m_n) + generation_prompt_ids

The invariant implementations aim for: that sum equals apply_chat_template(messages, add_generation_prompt=True) wherever the template is compositional across turns. Where a template is not compositional (e.g. templates that strip earlier-turn <think> blocks on full re-render), the incremental form is by definition what the model actually saw turn by turn, and is the recorded truth.

initial_ids: list[int]

Ids that open every conversation (BOS / template preamble); may be empty.

render_message(message: dict[str, Any]) -> list[int]

The ids for one fully framed turn (role header + content + turn separator).

generation_prompt_ids: list[int]

The ids that cue the model to answer (e.g. <|im_start|>assistant\n).

decode(token_ids: Sequence[int]) -> str

Decode sampled ids to text with special tokens stripped — the text the protocol parses.

sampled_suffix_ids: list[int]

Template ids that follow the stop token at the end of a sampled assistant turn.

A sampled turn’s ids run up to and including the stop token; a full-template render of that turn may append a separator after it (the \n after <|im_end|> in ChatML-family templates). Returning those ids lets a recorded turn replay as an exact prefix of a later context, matching what apply_chat_template produces.

Concrete with an empty default rather than abstract: an empty suffix is the correct answer for any frame that has none (most of them — Llama-3’s <|eot_id|> is already a turn’s last token), so a format only overrides when its template adds a separator.

Bases: BaseLLM

LLM backend that routes completions through GitHub Copilot.

Usage is charged to a Copilot subscription’s GitHub AI credits rather than to a per-vendor API key:

jaz.configure(llm=GitHubCopilotLLM(model="gpt-5-mini"))

Requires the jaz-lang[copilot] extra (github-copilot-sdk) and a GitHub Copilot subscription. Model ids are the ones copilot itself lists (the console’s setup wizard asks the account) and carry no provider/ prefix.

Sampling parameters (temperature, top_p, max_tokens) are rejected rather than silently dropped — the Copilot runtime owns the request body, and reasoning_effort is the one knob that maps through. See the module documentation for authentication order and the remaining limitations.

Parameters

  • model (str) — The Copilot model id, as copilot lists it (gpt-5-mini); no provider/ prefix.
  • base_directory (str | None) (default: None) — Where the Copilot runtime keeps its per-session state. Defaults to <jaz config dir>/copilot.
  • github_token (str | None) (default: None) — GitHub token to authenticate with. Defaults to the SDK’s own resolution (environment variables, then a signed-in CLI).
  • timeout (float) (default: 600.0) — Seconds to wait for one completion.
  • reuse_session (bool) (default: False) — Keep one Copilot session alive across calls instead of starting a fresh one per completion.
  • retry_policy (RetryPolicy | Mapping[str, Any]) (default: RetryPolicy()) — How failed calls are retried; see RetryPolicy. The Copilot CLI retries a failed model call itself first, so an error from a call it already retried is not retried again.
  • **query_params (Any) (default: {}) — per-request params the Copilot runtime accepts (reasoning_effort).

Raises

  • TypeError — A query param other than reasoning_effort is passed.
  • ValueError — model is empty, or timeout is not positive.
__init__(*, model: str, base_directory: str | None = None, github_token: str | None = None, timeout: float = 600.0, reuse_session: bool = False, retry_policy: RetryPolicy | Mapping[str, Any] = RetryPolicy(), **query_params: Any) -> None
complete(messages: list[Any]) -> LLMResponse
async acomplete(messages: list[Any]) -> LLMResponse
model_info: ModelInfo

The configured id; cost is reportable, as GitHub reports each call’s cost.

A turn whose figure is missing still reports cost_usd=None.

Bases: ChatFormat

A ChatFormat backed by a HuggingFace transformers tokenizer.

tokenizer is a model name/path (loaded via AutoTokenizer.from_pretrained — requires the jaz-lang[tokens] extra) or an already-constructed tokenizer object (no transformers import needed). Construction fails (ValueError) if the tokenizer has no chat_template, or if the template is not compositional per message (see below).

Limitations:

  • Templates that inject a default system prompt per turn (e.g. the Qwen2.5 and SmolLM2 families) are not compositional: rendering each message alone folds the default system block into every turn, so the incremental ids diverge from the real conversation. Such tokenizers are rejected at construction rather than silently recording ids the model never saw — there is no per-message workaround (an explicit system message does not suppress the per-turn re-injection). Use a template without per-turn default injection (e.g. Qwen3) or supply a custom ChatFormat.
  • Templates whose turn framing is plain text rather than special tokens (Vicuna-style USER:) may re-encode differently at the preamble seam; the compositionality is only guaranteed when the frame opens with a special token (ChatML, Llama-3, …).
  • Reasoning templates that drop earlier-turn <think> blocks on full re-render diverge from the incremental form by design; the incremental form is what the model actually saw (see ChatFormat).
__init__(tokenizer: str | Any) -> None
initial_ids: list[int]
render_message(message: dict[str, Any]) -> list[int]
generation_prompt_ids: list[int]
decode(token_ids: Sequence[int]) -> str
sampled_suffix_ids: list[int]

Normalized response from an LLM completion call.

The metric fields are named and defined as in ATIF v1.7 MetricsSchema:

  • prompt_tokens — total input tokens INCLUDING cached tokens
  • cached_tokens — subset of prompt_tokens that hit the cache
  • completion_tokens — total output tokens INCLUDING reasoning and tool-call tokens (matches OpenAI’s billing definition; for reasoning models the reasoning_tokens sub-count is surfaced in extra)
  • cost_usd — cost of this call in US dollars
  • extra — provider-specific metrics (e.g. reasoning_tokens, cache_creation_input_tokens for Anthropic), and where a backend reports them, ATIF’s optional prompt_token_ids, completion_token_ids and logprobs, and finish_reason (why the response stopped, in OpenAI’s vocabulary: "stop", "length", "tool_calls", …)

content and raw_response are not metrics and have no ATIF counterpart. There is no total_tokens: sum prompt_tokens and completion_tokens.

__init__(content: str | None, prompt_tokens: int | None = None, completion_tokens: int | None = None, cached_tokens: int | None = None, cost_usd: float | None = None, extra: dict[str, Any] = dict(), raw_response: Any = None, tokens: TurnRecord | None = None) -> None

Bases: BaseLLM

LLM backend that routes completions through LiteLLM — the default, reaching every provider LiteLLM routes to (OpenAI, Anthropic, Gemini, Bedrock, Vertex, …):

jaz.configure(llm=LiteLLM(model="openai/gpt-5-mini"))

The model id is LiteLLM’s routing key and is sent verbatim, so it carries a provider/ prefix (openai/gpt-5-mini, anthropic/claude-sonnet-5, vertex_ai/gemini-2.5-pro) written exactly as LiteLLM’s own documentation writes it. The provider LiteLLM routes to also selects the credential: a key stored for "openai" in ~/.jaz/credentials.json is used for openai/… models — and for a bare gpt-5-mini, since LiteLLM infers the provider when there is no prefix — behind an explicitly passed api_key and behind the provider’s environment variable. Cost is computed by litellm.completion_cost from LiteLLM’s price map, at the rates of the model that answered (and, for a region passed as aws_region_name, that region’s, where the map lists one), and reported only when model_info.can_report_cost (see ModelInfo), read from the configured model’s entry, says it can be. For a model id the map does not know, such as an Azure deployment name, pass LiteLLM’s base_model to name the model it serves; both cost and model_info then read that model’s entry:

LiteLLM(model="azure/my-deployment", base_model="azure/gpt-4o")

On OpenRouter, the cost is the amount OpenRouter reports charging, when it reports one, plus the provider’s charge to your own key when you bring one.

Parameters

  • model (str) — The LiteLLM route id, provider/model.
  • allowed_openai_params (list[str] | None) (default: None) — Request params LiteLLM would reject for this provider but should forward anyway (an escape hatch for a model newer than the installed LiteLLM).
  • drop_params (bool | None) (default: None) — Silently drop request params the provider does not support instead of raising.
  • retry_policy (RetryPolicy | Mapping[str, Any]) (default: RetryPolicy()) — How failed calls are retried; see RetryPolicy.
  • **query_params (Any) (default: {}) — Per-request defaults sent on every call (temperature, max_tokens, reasoning_effort, api_base, …).

Limitations

  • Importing litellm is slow (~2 s) and pulls a heavy dependency tree; the import is deferred until the first call, so import jaz does not pay it.
  • LiteLLM fetches its price map live, so reported cost can vary over time, and a model LiteLLM has not mapped reports no cost (model_info.can_report_cost is False for it).
  • A response that carries no token usage (an OpenAI-compatible server that omits usage, which LiteLLM fills in as zeros) reports its token counts and cost_usd as None. So does one reporting zero prompt tokens, which no real call has.
  • On anthropic/ models, extra["reasoning_tokens"] is LiteLLM’s estimate from the thinking text when Anthropic reports no count of its own.
  • A parameter LiteLLM recognizes is always transformed; allowed_openai_params forces it past validation but not past LiteLLM’s mapping.
  • LiteLLM’s fallbacks, model_list and context_window_fallback_dict are not supported, and neither is stream=True: passing any of them raises TypeError, as does LiteLLM’s num_retries (set retries with retry_policy), and a call made while litellm.model_fallbacks or litellm.num_retries is set raises UnsupportedParamsError.
  • A refused or filtered call raises ContentPolicyViolationError, and its cost is not reported.
  • A programming error inside LiteLLM (a KeyError, AttributeError, TypeError, …) is raised as LiteLLM’s own exception and is not retried, whether a bug in LiteLLM caused it or a response LiteLLM could not read, and even when LiteLLM hit it while handling a dropped connection. To retry it, add that exception’s class, such as litellm.APIConnectionError, to RetryPolicy.extra_retryable_exceptions.
__init__(*, model: str, allowed_openai_params: list[str] | None = None, drop_params: bool | None = None, retry_policy: RetryPolicy | Mapping[str, Any] = RetryPolicy(), **query_params: Any) -> None
complete(messages: list[Any]) -> LLMResponse
async acomplete(messages: list[Any]) -> LLMResponse
model_info: ModelInfo

The route id plus both facts from LiteLLM’s map (None/False if unmapped), read from base_model’s entry when it is set.

can_report_cost is True only when LiteLLM’s map publishes a per-token input rate for the model; a model priced another way (per second or per image, such as openai/sora-2) answers False, so a cost_budget on it is rejected up front.

Limitations: a published rate of 0 counts as one, so such a call is reported as free and a cost_budget on it never trips. And the first read is cached for the object’s lifetime: a model registered with LiteLLM after that read is not seen by this backend instance.

Bases: BaseLLM

A backend that answers from a callable you supply — for tests and scripted runs.

Every other backend talks to a provider; this one calls fn with the wire-form messages and returns what it answers, so an agent can be exercised with no API key, no network, and no nondeterminism. It is a full backend: it goes through jaz.configure and jaz.ConfigOverride like any other and reports cost, so BudgetPool and the other hooks see it exactly as they would a real one.

Retries are off by default (a raising callback is a failing test, not a flaky provider); pass retry_policy= to exercise the retry path itself.

fn receives the messages (the conversation in OpenAI chat format, in wire form) and returns either a str — the assistant’s reply, wrapped into an LLMResponse with zero token counts and zero cost — or an LLMResponse, used as is (set cost_usd yourself if a cost budget should count it). fn may be a coroutine function; acomplete awaits its result. It may raise: a jaz.exceptions LLM error is classified for retry like a provider’s, anything else propagates.

Example:

from jaz.llm import MockLLM
def scripted(messages):
last = messages[-1]["content"]
if "summarize" in last:
return "return 'summary'"
return "return 42"
with jaz.ConfigOverride(llm=MockLLM(scripted)):
assert jaz.invoke(task="add 40 and 2") == 42

A canned sequence is one lambda: replies = iter([...]); MockLLM(lambda m: next(replies)).

It serves no model, so model_info.name is None: observability hooks label nothing. No authored config can name it — it is constructed with a callable, which no config file can express — so passing the instance is the only way to select it.

__init__(fn: Callable[[list[Any]], str | LLMResponse], *, retry_policy: RetryPolicy | Mapping[str, Any] = RetryPolicy(max_retries=0)) -> None
complete(messages: list[Any]) -> LLMResponse
async acomplete(messages: list[Any]) -> LLMResponse
model_info: ModelInfo

What a backend knows about the model it serves; every field has an “unknown” value.

Attributes

  • name (str | None) — The model id, when the backend serves a nameable one — a routed id such as "openai/gpt-5-mini" for LiteLLM. None for a backend with no such notion (a mock, a server that serves whatever it was launched with). Observability hooks label spans with it when present and omit the label when not.
  • max_input_tokens (int | None) — The model’s input-token limit, or None when the backend cannot say. None means unknown, not unlimited — jaz.hooks.ContextWindowWarning disables itself on it rather than sizing a window from a guess.
  • can_report_cost (bool) — Whether this backend will set LLMResponse.cost_usd for this model. False is the honest default: the framework prices nothing itself. A cost budget on a backend that answers False is rejected before any token is spent.
__init__(name: str | None = None, max_input_tokens: int | None = None, can_report_cost: bool = False) -> None

How a backend retries a failed call: the backoff schedule and the retry caps.

Every count is a cap on extra retries, the LLM-SDK convention: max_retries=2 is up to three calls, 0 is no retry. A failed call is retried, with exponential backoff, while it is retryable and under every cap it falls under. It is retryable if it is an instance of a class in the backend’s retryable_exceptions (by default RateLimitError, ServerError and APIConnectionError, timeouts included) or in extra_retryable_exceptions. The provider can override that for one error (is_retryable, from its x-should-retry header), in either direction, so its “do not retry” stops even a class in extra_retryable_exceptions. max_retries caps every error, and each limits key caps the errors that are instances of its class.

Attributes

  • max_retries (int) — Cap on one call’s retries, in total across every error.
  • wait_multiplier (float) — Exponential-backoff multiplier, in seconds.
  • wait_min (float) — Shortest backoff between attempts, in seconds; jitter lengthens it at random by up to a quarter.
  • wait_max (float) — Cap on the exponential backoff, in seconds, before jitter lengthens it by up to a quarter. It does not cap a provider’s Retry-After; see retry_after_max.
  • retry_after_max (float) — Longest Retry-After a retry waits, in seconds. A provider’s Retry-After is waited instead of the backoff, plus up to a quarter of jitter; one longer than this stops the retries, and the call raises the error, whose retry_after says when to try again. 0 never waits one out.
  • limits (Mapping[Any, int]) — Caps on the retries of particular errors, keyed by exception class (RateLimitError, LLMError, a backend’s own class, …); any other key raises TypeError. Every key an error is an instance of caps it, and all errors under one key share its cap, so {LLMError: 3, RateLimitError: 1} allows rate limits 1 retry and all LLM errors 3 in total. A value above max_retries, or above the value of a key for one of its superclasses, raises ValueError, since it could never apply. A key only caps: an error that is not retryable fails at once, so a key on such a class applies only to errors the provider asked to retry. To give one error more retries than the rest of its family, raise max_retries and cap the siblings, not the family: max_retries=8, limits={ServerError: 2, APIConnectionError: 2} lets rate limits retry 8 times and every other retried error twice. A key on a class the backend maps to another before retrying never matches.
  • extra_retryable_exceptions (Collection[type[BaseException]]) — Exception classes to retry in addition to the backend’s retryable_exceptions, subclasses included, as a list, tuple or set (held as a tuple): extra_retryable_exceptions=[ContentPolicyViolationError] retries content-policy rejections, which can pass when resent, and [httpx.ReadTimeout] retries a timeout a backend did not map. A broad class retries everything under it: [Exception] retries every error a backend raises, bugs included, and [BaseException] also retries a task cancellation, Ctrl-C and sys.exit(), so none of them stops a call until its retries run out. Anything but exception classes raises TypeError. Like limits, it holds classes, so neither can be written in a config file.
__init__(max_retries: int = 10, wait_multiplier: float = 1.0, wait_min: float = 4.0, wait_max: float = 60.0, retry_after_max: float = 120.0, limits: Mapping[Any, int] = dict(), extra_retryable_exceptions: Collection[type[BaseException]] = ()) -> None

Bases: BaseLLM

Token-native LLM backend for SGLang’s /generate API.

Where the text backends send OpenAI-format messages and let the server own the chat template, this one renders the conversation to token ids itself and posts input_ids, receiving the sampled ids and per-token logprobs back:

jaz.configure(llm=SGLangLLM(model="Qwen/Qwen3-8B",
base_url="http://localhost:30000"))

Every response carries the exact tokens sent and sampled on LLMResponse.tokens, which is what makes agent runs recordable as training-grade rollouts.

chat_format defaults to HFChatFormat for model, which needs the jaz-lang[tokens] extra (HuggingFace transformers); pass a constructed ChatFormat to avoid that dependency. For models that emit an inline reasoning trace the trace is hidden from LLMResponse.content but kept in the token sidecar — see the module documentation for that and the remaining limitations.

Parameters

  • model (str) — The HuggingFace model name/path. Never sent — /generate serves whatever the server was launched with — it picks the default chat_format and labels the token record.
  • base_url (str) (default: 'http://localhost:30000') — Root URL of the SGLang server.
  • chat_format (ChatFormat | str | None) (default: None) — How to render messages to token ids. A ChatFormat, a model name/path for HFChatFormat, or None to derive one from model.
  • reasoning_tags (tuple[str, str] | list[str] | None) (default: ('<think>', '</think>')) — (open, close) delimiters of the hidden reasoning trace, or None to expose raw generations.
  • retry_policy (RetryPolicy | Mapping[str, Any]) (default: RetryPolicy()) — How failed calls are retried; see RetryPolicy.
  • **query_params (Any) (default: {}) — SGLang sampling params (temperature, max_new_tokens, …), max_tokens as an alias of max_new_tokens, and timeout, the HTTP timeout in seconds.

A request the server aborts (through /abort_request, a pause with mode="abort", or a weight update that aborts in-flight requests) raises RequestAbortedError, not retried by default, with whatever was generated in its body. Opting the class back in through RetryPolicy re-sends the original prompt from scratch on the policy’s backoff; it does not continue from the partial generation. SGLang answers a grammar failure the same way, so that raises it too. A request still waiting to reach the scheduler is aborted only on SGLang 0.5.17; on other releases it runs once generation resumes, possibly on new weights, so after a weight update check the response’s tokens.sampled.weight_version, which changes only when the update passes weight_version and names one version for the whole turn. A pause in mode="in_place" or mode="retract" holds the request open instead of aborting it; a timeout shorter than the pause turns it into an APITimeoutError, retried as a fresh request after the pause.

Raises

  • TypeError — A key in query_params is not a sampling param of SGLang 0.5.17; one added in a later SGLang is refused too.
  • ValueError — base_url is not an http:// or https:// URL with a host.
__init__(*, model: str, base_url: str = 'http://localhost:30000', chat_format: ChatFormat | str | None = None, reasoning_tags: tuple[str, str] | list[str] | None = ('<think>', '</think>'), retry_policy: RetryPolicy | Mapping[str, Any] = RetryPolicy(), **query_params: Any) -> None
complete(messages: list[Any]) -> LLMResponse
async acomplete(messages: list[Any]) -> LLMResponse
model_info: ModelInfo

The configured id only: no window, no cost.

Frozen token record for one message, carried on the message itself.

A caller finds them on a TurnRecord (LLMResponse.tokens) as sent[i] and sampled — the key a stamp rides under in the message mapping is an implementation detail and not part of the public API.

For a sampled assistant turn, ids is the message’s full contribution to a future context — the generation-prompt tokens that opened the turn plus the verbatim sampled ids (including the stop token) — and sampled_span marks the verbatim-sampled slice (the only part that gets loss mask 1). For any other message, ids is the chat format’s render of that one message and sampled_span is None.

Raises ValueError at construction when the sampled-only fields are inconsistent: sampled_span and logprobs must be present together and aligned, and weight_version/finish_reason are only meaningful on sampled stamps.

__init__(role: str, ids: tuple[int, ...], content_text: str, sampled_span: tuple[int, int] | None = None, logprobs: tuple[float, ...] | None = None, weight_version: str | None = None, finish_reason: str | None = None, message_id: str = (lambda: uuid.uuid4().hex)()) -> None

What one generation call sent and sampled. Built inside complete(); immutable.

sent holds one stamp per wire message, in order — shared references to the same objects that ride in the messages, not copies. sampled is the newly sampled assistant turn’s stamp. initial is the context preamble (BOS / template preamble) the backend prepended before any message’s ids; no message owns it.

Backend contract for sent that complete() must honor: exactly one stamp per wire message, in send order, with no merging of consecutive messages, reordering, or injected messages. The agent’s stamp-back pairs sent[i] with wire message i positionally, so any deviation misaligns the stamps (a length change raises; a same-length reorder or merge is caught per-message by role, else surfaces later as a broken export).

__init__(sent: tuple[TokenStamp, ...], sampled: TokenStamp, initial: tuple[int, ...] = ()) -> None