jaz.llm
New to JAZ? This is generated API reference. Start with the docs and the concepts overview.
Public API of the jaz.llm submodule (import from jaz.llm, e.g. from jaz.llm import ...).
The litellm backend routes through the LiteLLM library and is the default, reaching every
provider LiteLLM supports. GitHubCopilotLLM and SGLangLLM ship alongside it for the two cases it
does not serve well — completions billed to GitHub AI credits, and a self-hosted SGLang server
addressed token-in/token-out. MockLLM answers from a callable you supply, for deterministic
tests with no provider. Add your own by subclassing BaseLLM — see BaseLLM for
what a backend owns.
BaseLLM
Section titled “BaseLLM”Bases: ABC
One LLM backend: the API call, retry, and the cost and model metadata it reports.
Subclass this and pass an instance to add a backend:
class MyLLM(BaseLLM): def __init__(self, model, base_url, retry_policy=RetryPolicy(), **query_params): # retry_policy is named above, or the tail would take it super().__init__(retry_policy=retry_policy) self._model = model self._base_url = base_url self._query_params = query_params # yours; the base never sees them
def complete(self, messages) -> LLMResponse: wire = ... # call the API with self._model, messages, **self._query_params return LLMResponse(content=wire.text, prompt_tokens=..., extra={})LLMResponse is the whole return contract. With this class, and the two value types
RetryPolicy and ModelInfo, it is everything a backend author needs from
jaz.llm.
Declaring __init__ is all the config wiring a backend needs: its parameters are what
authored config can set, and a **query_params tail of its own is how it takes anything
it does not name. Give the class a __repr__ that renders that constructor call, with
every secret as '***': it is how the console’s settings view and a trace describe a
configured backend, shown verbatim and filtered by nothing else — so a dataclass-generated
repr would print an api_key field as is. Without a __repr__ they show the class
name and model_info.name. The backend stores what it was given and puts it on the request itself;
complete receives the messages and nothing else. Nothing passes through this class: a
backend that takes a tail names retry_policy explicitly and forwards only that.
To read a key from the shared credentials store (~/.jaz/credentials.json), call
jaz.credentials.resolve_credential with the provider’s tag — the default
LiteLLM backend does this per request, keyed by the provider it routes the model to.
A backend that never calls it does not read the store, so a key stored under its tag is
inert. Resolve it below an explicitly passed api_key and below the provider’s
environment variable, and hold it as your own attribute so the secret never becomes
construction config.
Retry is not a framework layer wrapped around backends: the policy is one construction
field of this base class, retry_policy (a RetryPolicy), and the framework runs
complete under it. That is what lets the config layer route it with no special
knowledge — it is discovered from this __init__ signature exactly as a backend’s
base_url is discovered from its own.
What is retried is declared by retryable_exceptions: a failed call is
retried only if it raised one of those classes, by default
RateLimitError, ServerError and
APIConnectionError (which includes
APITimeoutError). So raise a transient provider failure as one of
them, or add your own class to the tuple; anything else (a rejected request, a bug, a
cancellation) propagates at once, unless the RetryPolicy adds its class with
extra_retryable_exceptions. When the provider says how long to wait or whether to
retry, pass that on the error as retry_after (seconds) and is_retryable: the retry
follows the verdict and waits retry_after plus up to a quarter of jitter, except that a
retry_after longer than RetryPolicy.retry_after_max means no retry at all, whatever
the verdict.
The model is yours too. The base has no model: a backend that serves a nameable
model takes model= as its own constructor parameter and reports it through
model_info; one that does not (a fixed local server) has nothing to
declare.
What you know about the model is reported once, through model_info
(a ModelInfo): the name observability hooks label spans with, the input-token
limit ContextWindowWarning sizes its window from, and whether you set
LLMResponse.cost_usd — the framework prices nothing itself. Every field has an “unknown”
default, and each is honoured: a cost_budget on a backend that reports no cost is
rejected before any call rather than failing partway through, and an unknown window leaves
the context-window hook disabled rather than sized from a guess.
The one part that cannot be construction-time is the retry observer: the core loop passes an
on_retry callback carrying invoke_id/iteration to dispatch LLMQueryRetry,
which is per-call observability rather than policy.
BaseLLM.__init__
Section titled “BaseLLM.__init__”__init__(*, retry_policy: RetryPolicy | Mapping[str, Any] = RetryPolicy()) -> NoneBaseLLM.complete
Section titled “BaseLLM.complete”complete(messages: list[Any]) -> LLMResponseExecute a completion call against the LLM.
The messages are the only input. Everything else the call needs — the model, the sampling parameters, credentials — is the backend’s own state, set at construction and put on the request by this method.
Parameters
messages(list[Any]) — The conversation in OpenAI chat format, in wire form — the term used throughoutjazfor what actually goes to the provider: the agent’s message buffer with every reserved sidecar key stripped except those this backend lists inextra_message_keys.
Returns
LLMResponse— An LLMResponse with the completion content and usage information.
BaseLLM.acomplete
Section titled “BaseLLM.acomplete”async acomplete(messages: list[Any]) -> LLMResponseAsync version of complete(). Default: runs complete() in a thread.
Subclasses should override this for native async I/O.
BaseLLM.retryable_exceptions
Section titled “BaseLLM.retryable_exceptions”retryable_exceptions: tuple[type[BaseException], ...]Exception types a failed call is retried on by default; anything else is re-raised
at once, unless RetryPolicy.extra_retryable_exceptions adds it. The provider can
override this for one error (LLMError.is_retryable, from its x-should-retry
header).
The default is jaz’s transient classes: RateLimitError,
ServerError (a server error or invalid response) and
APIConnectionError (a failed transport, timeouts
included).
Override to add a class your backend raises for a transient failure, for example
(*super().retryable_exceptions, httpx.ReadTimeout).
BaseLLM.model_info
Section titled “BaseLLM.model_info”model_info: ModelInfoWhat this backend knows about its model — see ModelInfo.
The base knows nothing: no name, no input-token limit, no cost. Override to report what
yours can; LiteLLM answers from LiteLLM’s own model map.
ChatFormat
Section titled “ChatFormat”Bases: ABC
Renders chat messages to token ids, one message at a time.
A conversation’s generation input is assembled as:
initial_ids + render_message(m_1) + ... + render_message(m_n) + generation_prompt_idsThe invariant implementations aim for: that sum equals
apply_chat_template(messages, add_generation_prompt=True) wherever the template is
compositional across turns. Where a template is not compositional (e.g. templates
that strip earlier-turn <think> blocks on full re-render), the incremental form is
by definition what the model actually saw turn by turn, and is the recorded truth.
ChatFormat.initial_ids
Section titled “ChatFormat.initial_ids”initial_ids: list[int]Ids that open every conversation (BOS / template preamble); may be empty.
ChatFormat.render_message
Section titled “ChatFormat.render_message”render_message(message: dict[str, Any]) -> list[int]The ids for one fully framed turn (role header + content + turn separator).
ChatFormat.generation_prompt_ids
Section titled “ChatFormat.generation_prompt_ids”generation_prompt_ids: list[int]The ids that cue the model to answer (e.g. <|im_start|>assistant\n).
ChatFormat.decode
Section titled “ChatFormat.decode”decode(token_ids: Sequence[int]) -> strDecode sampled ids to text with special tokens stripped — the text the protocol parses.
ChatFormat.sampled_suffix_ids
Section titled “ChatFormat.sampled_suffix_ids”sampled_suffix_ids: list[int]Template ids that follow the stop token at the end of a sampled assistant turn.
A sampled turn’s ids run up to and including the stop token; a full-template
render of that turn may append a separator after it (the \n after <|im_end|>
in ChatML-family templates). Returning those ids lets a recorded turn replay as an
exact prefix of a later context, matching what apply_chat_template produces.
Concrete with an empty default rather than abstract: an empty suffix is the correct
answer for any frame that has none (most of them — Llama-3’s <|eot_id|> is already
a turn’s last token), so a format only overrides when its template adds a separator.
GitHubCopilotLLM
Section titled “GitHubCopilotLLM”Bases: BaseLLM
LLM backend that routes completions through GitHub Copilot.
Usage is charged to a Copilot subscription’s GitHub AI credits rather than to a per-vendor API key:
jaz.configure(llm=GitHubCopilotLLM(model="gpt-5-mini"))Requires the jaz-lang[copilot] extra (github-copilot-sdk) and a GitHub Copilot
subscription. Model ids are the ones copilot itself lists
(the console’s setup wizard asks the account) and carry no provider/ prefix.
Sampling parameters (temperature, top_p, max_tokens) are rejected rather than
silently dropped — the Copilot runtime owns the request body, and reasoning_effort is
the one knob that maps through. See the module documentation for authentication order and
the remaining limitations.
Parameters
model(str) — The Copilot model id, ascopilotlists it (gpt-5-mini); noprovider/prefix.base_directory(str | None) (default:None) — Where the Copilot runtime keeps its per-session state. Defaults to<jaz config dir>/copilot.github_token(str | None) (default:None) — GitHub token to authenticate with. Defaults to the SDK’s own resolution (environment variables, then a signed-in CLI).timeout(float) (default:600.0) — Seconds to wait for one completion.reuse_session(bool) (default:False) — Keep one Copilot session alive across calls instead of starting a fresh one per completion.retry_policy(RetryPolicy | Mapping[str, Any]) (default:RetryPolicy()) — How failed calls are retried; seeRetryPolicy. The Copilot CLI retries a failed model call itself first, so an error from a call it already retried is not retried again.**query_params(Any) (default:{}) — per-request params the Copilot runtime accepts (reasoning_effort).
Raises
TypeError— A query param other thanreasoning_effortis passed.ValueError—modelis empty, ortimeoutis not positive.
GitHubCopilotLLM.__init__
Section titled “GitHubCopilotLLM.__init__”__init__(*, model: str, base_directory: str | None = None, github_token: str | None = None, timeout: float = 600.0, reuse_session: bool = False, retry_policy: RetryPolicy | Mapping[str, Any] = RetryPolicy(), **query_params: Any) -> NoneGitHubCopilotLLM.complete
Section titled “GitHubCopilotLLM.complete”complete(messages: list[Any]) -> LLMResponseGitHubCopilotLLM.acomplete
Section titled “GitHubCopilotLLM.acomplete”async acomplete(messages: list[Any]) -> LLMResponseGitHubCopilotLLM.model_info
Section titled “GitHubCopilotLLM.model_info”model_info: ModelInfoThe configured id; cost is reportable, as GitHub reports each call’s cost.
A turn whose figure is missing still reports cost_usd=None.
HFChatFormat
Section titled “HFChatFormat”Bases: ChatFormat
A ChatFormat backed by a HuggingFace transformers tokenizer.
tokenizer is a model name/path (loaded via AutoTokenizer.from_pretrained —
requires the jaz-lang[tokens] extra) or an already-constructed tokenizer object (no
transformers import needed). Construction fails (ValueError) if the tokenizer
has no chat_template, or if the template is not compositional per message (see
below).
Limitations:
- Templates that inject a default system prompt per turn (e.g. the Qwen2.5 and
SmolLM2 families) are not compositional: rendering each message alone folds the
default system block into every turn, so the incremental ids diverge from the real
conversation. Such tokenizers are rejected at construction rather than silently
recording ids the model never saw — there is no per-message workaround (an explicit
system message does not suppress the per-turn re-injection). Use a template without
per-turn default injection (e.g. Qwen3) or supply a custom
ChatFormat. - Templates whose turn framing is plain text rather than special tokens (Vicuna-style
USER:) may re-encode differently at the preamble seam; the compositionality is only guaranteed when the frame opens with a special token (ChatML, Llama-3, …). - Reasoning templates that drop earlier-turn
<think>blocks on full re-render diverge from the incremental form by design; the incremental form is what the model actually saw (seeChatFormat).
HFChatFormat.__init__
Section titled “HFChatFormat.__init__”__init__(tokenizer: str | Any) -> NoneHFChatFormat.initial_ids
Section titled “HFChatFormat.initial_ids”initial_ids: list[int]HFChatFormat.render_message
Section titled “HFChatFormat.render_message”render_message(message: dict[str, Any]) -> list[int]HFChatFormat.generation_prompt_ids
Section titled “HFChatFormat.generation_prompt_ids”generation_prompt_ids: list[int]HFChatFormat.decode
Section titled “HFChatFormat.decode”decode(token_ids: Sequence[int]) -> strHFChatFormat.sampled_suffix_ids
Section titled “HFChatFormat.sampled_suffix_ids”sampled_suffix_ids: list[int]LLMResponse
Section titled “LLMResponse”Normalized response from an LLM completion call.
The metric fields are named and defined as in ATIF v1.7 MetricsSchema:
prompt_tokens— total input tokens INCLUDING cached tokenscached_tokens— subset ofprompt_tokensthat hit the cachecompletion_tokens— total output tokens INCLUDING reasoning and tool-call tokens (matches OpenAI’s billing definition; for reasoning models thereasoning_tokenssub-count is surfaced inextra)cost_usd— cost of this call in US dollarsextra— provider-specific metrics (e.g.reasoning_tokens,cache_creation_input_tokensfor Anthropic), and where a backend reports them, ATIF’s optionalprompt_token_ids,completion_token_idsandlogprobs, andfinish_reason(why the response stopped, in OpenAI’s vocabulary:"stop","length","tool_calls", …)
content and raw_response are not metrics and have no ATIF counterpart.
There is no total_tokens: sum prompt_tokens and completion_tokens.
LLMResponse.__init__
Section titled “LLMResponse.__init__”__init__(content: str | None, prompt_tokens: int | None = None, completion_tokens: int | None = None, cached_tokens: int | None = None, cost_usd: float | None = None, extra: dict[str, Any] = dict(), raw_response: Any = None, tokens: TurnRecord | None = None) -> NoneLiteLLM
Section titled “LiteLLM”Bases: BaseLLM
LLM backend that routes completions through LiteLLM — the default, reaching every provider LiteLLM routes to (OpenAI, Anthropic, Gemini, Bedrock, Vertex, …):
jaz.configure(llm=LiteLLM(model="openai/gpt-5-mini"))The model id is LiteLLM’s routing key and is sent verbatim, so it carries a provider/
prefix (openai/gpt-5-mini, anthropic/claude-sonnet-5, vertex_ai/gemini-2.5-pro)
written exactly as LiteLLM’s own documentation writes it. The provider LiteLLM routes to also
selects the credential: a key stored for "openai" in ~/.jaz/credentials.json is used
for openai/… models — and for a bare gpt-5-mini, since LiteLLM infers the provider
when there is no prefix — behind an explicitly passed api_key and behind the provider’s
environment variable. Cost is computed by litellm.completion_cost from LiteLLM’s
price map, at the rates of the model that answered (and, for a region passed as
aws_region_name, that region’s, where the map lists one), and reported only when model_info.can_report_cost (see ModelInfo), read
from the configured model’s entry, says it can be. For a model id the map does not know, such
as an Azure deployment name, pass LiteLLM’s base_model to name the model it serves; both
cost and model_info then read that model’s entry:
LiteLLM(model="azure/my-deployment", base_model="azure/gpt-4o")On OpenRouter, the cost is the amount OpenRouter reports charging, when it reports one, plus the provider’s charge to your own key when you bring one.
Parameters
model(str) — The LiteLLM route id,provider/model.allowed_openai_params(list[str] | None) (default:None) — Request params LiteLLM would reject for this provider but should forward anyway (an escape hatch for a model newer than the installed LiteLLM).drop_params(bool | None) (default:None) — Silently drop request params the provider does not support instead of raising.retry_policy(RetryPolicy | Mapping[str, Any]) (default:RetryPolicy()) — How failed calls are retried; seeRetryPolicy.**query_params(Any) (default:{}) — Per-request defaults sent on every call (temperature,max_tokens,reasoning_effort,api_base, …).
Limitations
- Importing
litellmis slow (~2 s) and pulls a heavy dependency tree; the import is deferred until the first call, soimport jazdoes not pay it.- LiteLLM fetches its price map live, so reported cost can vary over time, and a model LiteLLM has not mapped reports no cost (
model_info.can_report_costisFalsefor it).- A response that carries no token usage (an OpenAI-compatible server that omits
usage, which LiteLLM fills in as zeros) reports its token counts andcost_usdasNone. So does one reporting zero prompt tokens, which no real call has.- On
anthropic/models,extra["reasoning_tokens"]is LiteLLM’s estimate from the thinking text when Anthropic reports no count of its own.- A parameter LiteLLM recognizes is always transformed;
allowed_openai_paramsforces it past validation but not past LiteLLM’s mapping.- LiteLLM’s
fallbacks,model_listandcontext_window_fallback_dictare not supported, and neither isstream=True: passing any of them raisesTypeError, as does LiteLLM’snum_retries(set retries withretry_policy), and a call made whilelitellm.model_fallbacksorlitellm.num_retriesis set raisesUnsupportedParamsError.- A refused or filtered call raises
ContentPolicyViolationError, and its cost is not reported.- A programming error inside LiteLLM (a
KeyError,AttributeError,TypeError, …) is raised as LiteLLM’s own exception and is not retried, whether a bug in LiteLLM caused it or a response LiteLLM could not read, and even when LiteLLM hit it while handling a dropped connection. To retry it, add that exception’s class, such aslitellm.APIConnectionError, toRetryPolicy.extra_retryable_exceptions.
LiteLLM.__init__
Section titled “LiteLLM.__init__”__init__(*, model: str, allowed_openai_params: list[str] | None = None, drop_params: bool | None = None, retry_policy: RetryPolicy | Mapping[str, Any] = RetryPolicy(), **query_params: Any) -> NoneLiteLLM.complete
Section titled “LiteLLM.complete”complete(messages: list[Any]) -> LLMResponseLiteLLM.acomplete
Section titled “LiteLLM.acomplete”async acomplete(messages: list[Any]) -> LLMResponseLiteLLM.model_info
Section titled “LiteLLM.model_info”model_info: ModelInfoThe route id plus both facts from LiteLLM’s map (None/False if unmapped),
read from base_model’s entry when it is set.
can_report_cost is True only when LiteLLM’s map publishes a per-token input
rate for the model; a model priced another way (per second or per image, such as
openai/sora-2) answers False, so a cost_budget on it is rejected up front.
Limitations: a published rate of 0 counts as one, so such a call is reported as
free and a cost_budget on it never trips. And the first read is cached for the
object’s lifetime: a model registered with LiteLLM after that read is not seen by this
backend instance.
MockLLM
Section titled “MockLLM”Bases: BaseLLM
A backend that answers from a callable you supply — for tests and scripted runs.
Every other backend talks to a provider; this one calls fn with the wire-form
messages and returns what it answers, so an agent can be exercised with no API key, no
network, and no nondeterminism. It is a full backend: it goes through jaz.configure
and jaz.ConfigOverride like any other and reports cost, so BudgetPool and the other
hooks see it exactly as they would a real one.
Retries are off by default (a raising callback is a failing test, not a flaky provider);
pass retry_policy= to exercise the retry path itself.
fn receives the messages (the conversation in OpenAI chat format, in wire form) and
returns either a str — the assistant’s reply, wrapped into an LLMResponse
with zero token counts and zero cost — or an LLMResponse, used as is (set
cost_usd yourself if a cost budget should count it). fn may be a coroutine
function; acomplete awaits its result. It may raise: a jaz.exceptions LLM error
is classified for retry like a provider’s, anything else propagates.
Example:
from jaz.llm import MockLLM
def scripted(messages): last = messages[-1]["content"] if "summarize" in last: return "return 'summary'" return "return 42"
with jaz.ConfigOverride(llm=MockLLM(scripted)): assert jaz.invoke(task="add 40 and 2") == 42A canned sequence is one lambda: replies = iter([...]); MockLLM(lambda m: next(replies)).
It serves no model, so model_info.name is None: observability hooks label nothing.
No authored config can name it — it is constructed with a callable, which no config file
can express — so passing the instance is the only way to select it.
MockLLM.__init__
Section titled “MockLLM.__init__”__init__(fn: Callable[[list[Any]], str | LLMResponse], *, retry_policy: RetryPolicy | Mapping[str, Any] = RetryPolicy(max_retries=0)) -> NoneMockLLM.complete
Section titled “MockLLM.complete”complete(messages: list[Any]) -> LLMResponseMockLLM.acomplete
Section titled “MockLLM.acomplete”async acomplete(messages: list[Any]) -> LLMResponseMockLLM.model_info
Section titled “MockLLM.model_info”model_info: ModelInfoModelInfo
Section titled “ModelInfo”What a backend knows about the model it serves; every field has an “unknown” value.
Attributes
name(str | None) — The model id, when the backend serves a nameable one — a routed id such as"openai/gpt-5-mini"forLiteLLM.Nonefor a backend with no such notion (a mock, a server that serves whatever it was launched with). Observability hooks label spans with it when present and omit the label when not.max_input_tokens(int | None) — The model’s input-token limit, orNonewhen the backend cannot say.Nonemeans unknown, not unlimited —jaz.hooks.ContextWindowWarningdisables itself on it rather than sizing a window from a guess.can_report_cost(bool) — Whether this backend will setLLMResponse.cost_usdfor this model.Falseis the honest default: the framework prices nothing itself. A cost budget on a backend that answersFalseis rejected before any token is spent.
ModelInfo.__init__
Section titled “ModelInfo.__init__”__init__(name: str | None = None, max_input_tokens: int | None = None, can_report_cost: bool = False) -> NoneRetryPolicy
Section titled “RetryPolicy”How a backend retries a failed call: the backoff schedule and the retry caps.
Every count is a cap on extra retries, the LLM-SDK convention: max_retries=2 is up to
three calls, 0 is no retry. A failed call is retried, with exponential backoff, while
it is retryable and under every cap it falls under. It is retryable if it is an instance of
a class in the backend’s retryable_exceptions (by default
RateLimitError, ServerError and
APIConnectionError, timeouts included) or in
extra_retryable_exceptions. The provider can override that for one error
(is_retryable, from its x-should-retry header), in
either direction, so its “do not retry” stops even a class in
extra_retryable_exceptions.
max_retries caps every error, and each limits key caps the errors that are
instances of its class.
Attributes
max_retries(int) — Cap on one call’s retries, in total across every error.wait_multiplier(float) — Exponential-backoff multiplier, in seconds.wait_min(float) — Shortest backoff between attempts, in seconds; jitter lengthens it at random by up to a quarter.wait_max(float) — Cap on the exponential backoff, in seconds, before jitter lengthens it by up to a quarter. It does not cap a provider’sRetry-After; seeretry_after_max.retry_after_max(float) — LongestRetry-Aftera retry waits, in seconds. A provider’sRetry-Afteris waited instead of the backoff, plus up to a quarter of jitter; one longer than this stops the retries, and the call raises the error, whoseretry_aftersays when to try again.0never waits one out.limits(Mapping[Any, int]) — Caps on the retries of particular errors, keyed by exception class (RateLimitError,LLMError, a backend’s own class, …); any other key raisesTypeError. Every key an error is an instance of caps it, and all errors under one key share its cap, so{LLMError: 3, RateLimitError: 1}allows rate limits 1 retry and all LLM errors 3 in total. A value abovemax_retries, or above the value of a key for one of its superclasses, raisesValueError, since it could never apply. A key only caps: an error that is not retryable fails at once, so a key on such a class applies only to errors the provider asked to retry. To give one error more retries than the rest of its family, raisemax_retriesand cap the siblings, not the family:max_retries=8, limits={ServerError: 2, APIConnectionError: 2}lets rate limits retry 8 times and every other retried error twice. A key on a class the backend maps to another before retrying never matches.extra_retryable_exceptions(Collection[type[BaseException]]) — Exception classes to retry in addition to the backend’sretryable_exceptions, subclasses included, as a list, tuple or set (held as a tuple):extra_retryable_exceptions=[ContentPolicyViolationError]retries content-policy rejections, which can pass when resent, and[httpx.ReadTimeout]retries a timeout a backend did not map. A broad class retries everything under it:[Exception]retries every error a backend raises, bugs included, and[BaseException]also retries a task cancellation, Ctrl-C andsys.exit(), so none of them stops a call until its retries run out. Anything but exception classes raisesTypeError. Likelimits, it holds classes, so neither can be written in a config file.
RetryPolicy.__init__
Section titled “RetryPolicy.__init__”__init__(max_retries: int = 10, wait_multiplier: float = 1.0, wait_min: float = 4.0, wait_max: float = 60.0, retry_after_max: float = 120.0, limits: Mapping[Any, int] = dict(), extra_retryable_exceptions: Collection[type[BaseException]] = ()) -> NoneSGLangLLM
Section titled “SGLangLLM”Bases: BaseLLM
Token-native LLM backend for SGLang’s /generate API.
Where the text backends send OpenAI-format messages and let the server own the chat
template, this one renders the conversation to token ids itself and posts input_ids,
receiving the sampled ids and per-token logprobs back:
jaz.configure(llm=SGLangLLM(model="Qwen/Qwen3-8B", base_url="http://localhost:30000"))Every response carries the exact tokens sent and sampled on LLMResponse.tokens, which is
what makes agent runs recordable as training-grade rollouts.
chat_format defaults to HFChatFormat for model, which needs the
jaz-lang[tokens] extra (HuggingFace transformers); pass a constructed
ChatFormat to avoid that dependency. For models that emit an inline reasoning
trace the trace is hidden from LLMResponse.content but kept in the token sidecar — see
the module documentation for that and the remaining limitations.
Parameters
model(str) — The HuggingFace model name/path. Never sent —/generateserves whatever the server was launched with — it picks the defaultchat_formatand labels the token record.base_url(str) (default:'http://localhost:30000') — Root URL of the SGLang server.chat_format(ChatFormat | str | None) (default:None) — How to render messages to token ids. AChatFormat, a model name/path forHFChatFormat, orNoneto derive one frommodel.reasoning_tags(tuple[str, str] | list[str] | None) (default:('<think>', '</think>')) —(open, close)delimiters of the hidden reasoning trace, orNoneto expose raw generations.retry_policy(RetryPolicy | Mapping[str, Any]) (default:RetryPolicy()) — How failed calls are retried; seeRetryPolicy.**query_params(Any) (default:{}) — SGLang sampling params (temperature,max_new_tokens, …),max_tokensas an alias ofmax_new_tokens, andtimeout, the HTTP timeout in seconds.
A request the server aborts (through /abort_request, a pause with mode="abort", or
a weight update that aborts in-flight requests) raises
RequestAbortedError, not retried by default, with whatever was
generated in its body. Opting the class back in through RetryPolicy re-sends the
original prompt from scratch on the policy’s backoff; it does not continue from the partial
generation. SGLang answers a grammar failure the same way, so that raises it
too. A request still waiting to reach the scheduler is aborted only on SGLang 0.5.17; on
other releases it runs once generation resumes, possibly on new weights, so after a weight
update check the response’s tokens.sampled.weight_version, which changes only when the
update passes weight_version and names one version for the whole turn. A pause in
mode="in_place" or mode="retract" holds the request open instead of aborting it; a
timeout shorter than the pause turns it into an APITimeoutError, retried as a fresh
request after the pause.
Raises
TypeError— A key inquery_paramsis not a sampling param of SGLang 0.5.17; one added in a later SGLang is refused too.ValueError—base_urlis not anhttp://orhttps://URL with a host.
SGLangLLM.__init__
Section titled “SGLangLLM.__init__”__init__(*, model: str, base_url: str = 'http://localhost:30000', chat_format: ChatFormat | str | None = None, reasoning_tags: tuple[str, str] | list[str] | None = ('<think>', '</think>'), retry_policy: RetryPolicy | Mapping[str, Any] = RetryPolicy(), **query_params: Any) -> NoneSGLangLLM.complete
Section titled “SGLangLLM.complete”complete(messages: list[Any]) -> LLMResponseSGLangLLM.acomplete
Section titled “SGLangLLM.acomplete”async acomplete(messages: list[Any]) -> LLMResponseSGLangLLM.model_info
Section titled “SGLangLLM.model_info”model_info: ModelInfoThe configured id only: no window, no cost.
TokenStamp
Section titled “TokenStamp”Frozen token record for one message, carried on the message itself.
A caller finds them on a TurnRecord (LLMResponse.tokens) as
sent[i] and sampled — the key a stamp rides under in the message mapping is an
implementation detail and not part of the public API.
For a sampled assistant turn, ids is the message’s full contribution to a future
context — the generation-prompt tokens that opened the turn plus the verbatim sampled
ids (including the stop token) — and sampled_span marks the verbatim-sampled slice
(the only part that gets loss mask 1). For any other message, ids is the chat
format’s render of that one message and sampled_span is None.
Raises ValueError at construction when the sampled-only fields are inconsistent:
sampled_span and logprobs must be present together and aligned, and
weight_version/finish_reason are only meaningful on sampled stamps.
TokenStamp.__init__
Section titled “TokenStamp.__init__”__init__(role: str, ids: tuple[int, ...], content_text: str, sampled_span: tuple[int, int] | None = None, logprobs: tuple[float, ...] | None = None, weight_version: str | None = None, finish_reason: str | None = None, message_id: str = (lambda: uuid.uuid4().hex)()) -> NoneTurnRecord
Section titled “TurnRecord”What one generation call sent and sampled. Built inside complete(); immutable.
sent holds one stamp per wire message, in order — shared references to the same
objects that ride in the messages, not copies. sampled is the newly sampled
assistant turn’s stamp. initial is the context preamble (BOS / template preamble)
the backend prepended before any message’s ids; no message owns it.
Backend contract for sent that complete() must honor: exactly one stamp per wire
message, in send order, with no merging of consecutive messages, reordering, or injected
messages. The agent’s stamp-back pairs sent[i] with wire message i positionally,
so any deviation misaligns the stamps (a length change raises; a same-length reorder or
merge is caught per-message by role, else surfaces later as a broken export).
TurnRecord.__init__
Section titled “TurnRecord.__init__”__init__(sent: tuple[TokenStamp, ...], sampled: TokenStamp, initial: tuple[int, ...] = ()) -> None