Skip to content

API reference

Generated from the docstrings of the public names in llm_queue.__all__.

llm_queue

llm-queue: LLM calls through a queue, balanced across accounts and regions.

This is the public front door. LLMQueue (loaded on first use, so importing the package has no side effects) runs calls through each model's queue under RPM / TPM budgets per deployment (model x account x region), with fallback regions and models, health and status. The registry (LLMQ_REGISTRY) describes accounts, models and settings; the extension points are plain functions registered by name.

RequestContext dataclass

RequestContext(priority: bool = False, session_id: str | None = None, user_id: str | None = None, tenant_id: str | None = None)

What the current request is: its class and the ids used for sticky sessions and records.

CallFailed

CallFailed(kind: Kind | str, message: str = '', *, retry_after_secs: float | None = None, partial_text: str | None = None, request_id: str | None = None)

Bases: LLMQueueError

A call ended without a reply; kind is one of KINDS.

Keep the kind and its details.

CallContext dataclass

CallContext(model: str, request_id: str, priority: bool, session_id: str | None, user_id: str | None, tenant_id: str | None, waited_secs: float = 0.0)

What hooks see of a call (no prompt text).

DeploymentView dataclass

DeploymentView(key: str, account: str, region: str, group: str, rpm: int | None, tpm: int | None, weight: float, tags: Mapping[str, Any] = dict())

What a custom balancer sees of one candidate deployment.

Reject dataclass

Reject(reason: str)

Returned by an admission hook to refuse a call before it is sent.

ConfigError

Bases: LLMQueueError

Invalid configuration, raised at startup or on a refused edit (never mid-call).

LLMQueueError

Bases: Exception

Base class for every error this package raises to callers.

MissingExtra

MissingExtra(feature: str, extra: str)

Bases: ConfigError, ImportError

An optional feature's dependencies are not installed; names the extra to install.

Also an ImportError, so except ImportError around optional imports keeps working.

feature needs pip install 'llm-queue[extra]'.

StoreUnavailable

Bases: LLMQueueError, ConnectionError

The state store could not serve an admin operation (unreachable, read-only, full).

Raised by cancel, status, send_job and the registry's edits and reads; a call raises CallFailed instead. Also a ConnectionError, so existing handlers keep working.

UnknownModel

Bases: ConfigError, KeyError

A model name that is neither an alias nor a model id in the registry.

Also a KeyError, so code that looks models up by key keeps working (HTTP 404, CLI).

BedrockOptions dataclass

BedrockOptions(read_timeout_secs: float = 300.0, connect_timeout_secs: float = 10.0, max_pool_connections: int = 100, role_session_secs: int = 3600, transport: str = 'aiohttp')

The Bedrock and STS clients (clients/pool.py).

EngineTimings dataclass

EngineTimings(idle_poll_secs: float = 0.05, idle_slow_secs: float = 1.0, idle_after_empty: int = 20, result_recheck_secs: float = 60.0, health_probe_secs: float = 3.0, stream_read_block_ms: int = 100, admission_probe_wait_ms: int = 200, admission_slot_wait_ms: int = 50, admission_slot_jitter_ms: int = 150, admission_wait_ms: int = 100, limiter_cut_ms: int = 1000, limiter_cut_factor: float = 0.7, budget_key_ms: int = 180000, throttle_base_ms: int = 1000, throttle_max_ms: int = 30000, slow_stale_ms: int = 300000, job_ttl_ms: int = 3600000)

Pacing of the engine's loops (engine/runtime.py, engine/streaming.py).

OpenAIOptions dataclass

OpenAIOptions(read_timeout_secs: float = 300.0, connect_timeout_secs: float = 10.0, max_connections: int = 100, max_keepalive_connections: int = 20, keepalive_secs: float = 15.0, http: Literal['httpx', 'aiohttp'] = 'aiohttp')

The OpenAI-compatible clients' timeouts, connection pool and HTTP client (one per account).

http="aiohttp" sends through the SDK's aiohttp client with the same limits and timeouts; aiohttp has no idle cap, so max_keepalive_connections is not used there. Without one (openai older than 1.9x, or no httpx-aiohttp / aiohttp), httpx is used and a warning logged.

RegistryOptions dataclass

RegistryOptions(history_size: int = 20, edit_retries: int = 3)

The registry store (registry/store.py).

Audit retention stays LLMQ_AUDIT_RETENTION_DAYS.

StoreOptions dataclass

StoreOptions(retries: int = 3, connect_timeout_secs: float = 5.0, read_timeout_secs: float = 15.0, health_check_secs: float = 30.0, pool_wait_secs: float = 10.0)

The Valkey / Redis client of every process (engine/store_client.py).

The pool size stays LLMQ_STORE_MAX_CONNECTIONS: it follows the deployment's slots.

Tuning dataclass

Tuning(store: StoreOptions = StoreOptions(), bedrock: BedrockOptions = BedrockOptions(), engine: EngineTimings = EngineTimings(), registry: RegistryOptions = RegistryOptions(), openai: OpenAIOptions = OpenAIOptions(), warm: WarmOptions = WarmOptions())

All of the above; Tuning() is the defaults. Passed to LLMQueue.from_env(tuning=).

problems

problems(*, queue_wait_ms: int = 0, result_ttl_secs: int | None = None) -> list[str]

Every bad value and bad combination, one line each (empty when all is well).

queue_wait_ms is LLMQ_QUEUE_WAIT_MS: WAIT blocks a connection that long, so the store read timeout must be above it. result_ttl_secs is LLMQ_RESULT_TTL_SECS: a result must outlive a waiting caller's last-resort read (R3CF-10).

WarmOptions dataclass

WarmOptions(connections: int = 2, refresh_secs: float = 10.0, keepalive_secs: float = 15.0)

Warm connections per Bedrock client (E4); pass connections=0 to turn it off.

After a client is built, connections unsigned HEAD requests go to its endpoint at once through the client's own aiohttp session, so DNS, TCP and TLS are done before the first call. They are not billable (no model is invoked; AWS answers an unsigned request with an error status and keeps the connection). Every refresh_secs the same requests run again, so the connections outlive AWS's ~20 s idle close and aiohttp's keepalive_secs.

Route dataclass

Route(account: Account, region: str, model_id: str, deployment: str)

Where one attempt goes: account, region to call, model id, and the deployment's name.

Provider

Bases: Protocol

One provider kind: runs an operation on a route and classifies its failures.

call async

call(op: str, params: Mapping[str, Any], route: Route, timing: Timing, sink: Sink | None = None, *, converse_api: str = 'converse') -> Result

Run one attempt; chat operations pass each streamed event to sink.

converse_api is the model's setting: converse or stream (Bedrock only).

classify

classify(exc: BaseException) -> ErrorInfo

Throttle / retryable / fatal.

ConcurrencyStatus

Bases: TypedDict

Calls in flight on one deployment and its (adaptive) limit.

DeploymentStatus

Bases: TypedDict

One deployment of a model: switches, budgets, concurrency, breaker and latency.

account_enabled and blocked (a routing block on its region or account) were returned but not declared, so a type checker refused reading them (R3AP-3).

EmbedResult dataclass

EmbedResult(vectors: list[list[float]], meta: Meta, errors: dict[int, CallFailed] = dict(), batches: int = 0, request_ids: list[str] = list())

Vectors (one per text, in order) and how the batches were served.

With embed(partial=True), a text that failed has the vector [] and its error in errors (by text index); otherwise errors is always empty. meta.tokens_in sums the input tokens the replies reported; batches counts provider calls and request_ids lists the answered ones, for cost and support (Q21).

Health

Bases: TypedDict

Liveness, readiness, each store's state (ok, read_only, down) and eviction policy.

eviction_policy is the state store's maxmemory-policy read at start (unknown when unreadable); queue_eviction_policy is the queue store's, when it is a separate store.

LLMQueue

LLMQueue(engine: Engine, *, settings: InfraSettings | None = None, builder: Callable[[], Engine] | None = None, offload: ResultOffload | None = None, job_settings: JobSettings | None = None, tuning: Tuning | None = None)

One process's LLM queue: caller, worker, jobs and admin over the stores.

Wrap a built Engine; builder makes a caller-only engine for sync calls.

That engine is used only while this queue is not started (sync-only processes). job_settings (default: LLMQ_JOB_* read by from_env) bounds jobs. tuning is the one the engine was built with (from_env passes it): start() compares its client timeouts with the registry (R3CF-10).

engine property

engine: Engine

Deprecated: the engine is private (owner decision Q35); use the queue's methods.

Kept as an alias that warns until 0.2.

jobs property

jobs: JobRunner

Deprecated: the job runner is private (Q35); use register_job and send_job.

registry property

registry: RegistryStore

The registry store (live edits, audit, rollback).

from_env classmethod

from_env(environ: Mapping[str, str] | None = None, *, secrets: Secrets | None = None, metrics: Metrics | None = None, worker_id: str | None = None, credentials: Callable[[Any], Mapping[str, Any] | None] | None = None, offload: ResultOffload | None = None, offload_bytes: int | None = None, tuning: Tuning | None = None) -> LLMQueue

Build from LLMQ_* (secret first when secrets is given, then the environment).

tuning sets timeouts, pool sizes and loop pacing in code (Tuning() is the defaults); a bad value or combination raises ConfigError listing every problem.

start async

start() -> None

Publish / load the registry; run the worker side unless mode=caller.

A second start() does nothing; a stopped queue cannot start again (build a new one). The first started queue of the process is the LangChain default and stays so. Starts run one at a time, on one event loop. A start that fails part-way stops what it started and re-raises; the queue is then stopped (a stopped engine cannot start again), and a later start() raises LLMQueueError naming that first error.

pause_receiving

pause_receiving() -> None

Fail readiness and take no new calls or jobs; running work goes on (preStop hook).

stop async

stop(drain_secs: float | None = None) -> None

Shut down in order (plan ยง3.18) and close every client; a second call does nothing.

drain_secs (default LLMQ_DRAIN_SECS) bounds the whole wait: jobs first (they make calls), then running calls get what is left of it. Calls made during the drain are served; new calls are refused once it is over. Every step runs even when an earlier one fails (each failure is logged) and the first failure is raised at the end; if the stop is cancelled, the steps left still close their clients, without draining.

call async

call(model: str, params: Mapping[str, Any], *, priority: bool | None = None, **kw: Any) -> Reply

Any operation of a model (converse, chat, invoke_model, embeddings).

converse, chat and call run the same code: the model's provider, not the method name, decides the request shape (Converse for Bedrock, chat completions for OpenAI-compatible). Use call for code that handles every model alike.

Per-call options (keyword only; R3AP-10): timeout_secs (default the model's timeout_secs, capped by timeout_max_secs), max_wait_secs (longest wait for budget), request_id (your id; a repeat attaches to the same call), session_id, user_id, tenant_id (default from request_context), allow_fallback ("all": fallback regions and models, "regions": regions only, "none"), allowed_regions and allowed_accounts (limit the deployments). An unknown option raises TypeError.

converse async

converse(model: str, request: Mapping[str, Any], *, priority: bool | None = None, **kw: Any) -> Reply

One chat call, named for Bedrock: send a Converse request to a Bedrock model.

Same options and code as call; the reply's response is the Converse body.

chat async

chat(model: str, request: Mapping[str, Any], *, priority: bool | None = None, **kw: Any) -> Reply

One chat call, named for OpenAI: send a chat completion to an OpenAI-compatible model.

Same options and code as call; the reply's response is the completion body.

converse_stream async

converse_stream(model: str, request: Mapping[str, Any], *, priority: bool | None = None, **kw: Any) -> AsyncIterator[dict[str, Any]]

Stream a chat call's events as they arrive (plan ยง3.13).

Bedrock models yield ConverseStream events, OpenAI-compatible ones chat completion chunks. A failure after events were yielded raises CallFailed with partial_text.

chat_stream async

chat_stream(model: str, request: Mapping[str, Any], *, priority: bool | None = None, **kw: Any) -> AsyncIterator[dict[str, Any]]

converse_stream under the OpenAI name: chat completion chunks from such a model.

Was an alias, so help() and tracebacks showed converse_stream (R3AP-15).

embed async

embed(model: str, texts: Sequence[str], *, priority: bool | None = None, input_type: str | None = None, partial: bool = False, **kw: Any) -> EmbedResult

One vector per text, in order (batched by the model's embed_batch_size).

partial=True keeps the batches that succeeded (see EmbedResult.errors); by default one failed batch fails the call. request_id=X names batch n X:n.

converse_sync

converse_sync(model: str, request: Mapping[str, Any], *, priority: bool | None = None, **kw: Any) -> Reply

converse on the direct path from sync code (same budgets, no queue).

chat_sync

chat_sync(model: str, request: Mapping[str, Any], *, priority: bool | None = None, **kw: Any) -> Reply

chat on the direct path from sync code.

embed_sync

embed_sync(model: str, texts: Sequence[str], *, priority: bool | None = None, input_type: str | None = None, partial: bool = False, **kw: Any) -> EmbedResult

embed on the direct path from sync code.

close_sync

close_sync() -> None

Stop the private engine of the sync calls (sync-only processes; stop() does too).

register_job

register_job(name: str, fn: JobFn, *, timeout_secs: float | None = None, concurrency: int | None = None) -> None

Allow this process to run job name (call before start).

concurrency caps its runs in this process (default LLMQ_JOB_CONCURRENCY, 4).

A run longer than timeout_secs is cancelled and counts as a failure (retried, then dead-lettered); None (the default) lets it run as long as it takes. ConfigError in caller mode, which runs no jobs: the job waited unseen (R3JE-6).

send_job async

send_job(name: str, payload: Mapping[str, Any], *, job_id: str | None = None) -> str

Queue a job; returns its id (jobs run at least once).

A second send with the same job_id within LLMQ_JOB_DEDUPE_SECS (24 h) queues nothing and returns the id. CallFailed("queue_full") when 10,000 jobs of this name already wait; request_too_large above LLMQ_JOB_MAX_PAYLOAD_BYTES (1 MB); invalid_request for a payload that is not JSON.

job_dead_letters async

job_dead_letters(name: str, count: int = 10) -> list[dict[str, Any]]

Dead jobs of one name, oldest first: entry id, job id, failures, payload (Q15).

redrive_jobs async

redrive_jobs(name: str, count: int = 1000) -> int

Send up to count dead jobs of one name back to run again; returns how many.

cancel async

cancel(request_id: str) -> None

Cancel a call (waiting or running).

status async

status(model: None = None) -> StatusReport
status(model: str) -> ModelStatus
status(model: str | None = None) -> StatusReport | ModelStatus

Status of a model (ModelStatus) or of everything (StatusReport).

request_status async

request_status(request_id: str) -> RequestStatus

Status of one call (empty when unknown).

health async

health() -> Health

Liveness, readiness and the stores' state.

http_router

http_router(*, dependencies: Sequence[Any] = (), allow_writes: bool = False, actor: Callable[[Any], str] | None = None) -> Any

A FastAPI router with status, requests, registry and health (extra http).

Writes need actor(request) -> str, the authenticated name recorded in the audit.

ModelStatus

Bases: TypedDict

One model: waiting calls per class, calls in progress, dead letters, deployments.

RateStatus

Bases: TypedDict

A budget of one deployment this minute.

RequestStatus

Bases: TypedDict

One call's status record as stored (empty when the id is unknown).

Every value is text, as the store keeps it: convert the numbers with int() (attempts, tokens_in, tokens_out) and the *_at times (Unix seconds) with float(). One *_at key is there per state the call reached (R3AP-4).

StatusReport

Bases: TypedDict

Every model and job, the registry version and the routing switches.

Meta

Bases: BaseModel

How a call was served: deployment, fallback, attempts, timings, tokens.

Reply

Bases: BaseModel

The stored result of a call: the provider reply, or an error kind with its details.

dumps

dumps() -> bytes

JSON bytes for the store (fields directly: model_dump re-walked the response).

JobContext dataclass

JobContext(name: str, job_id: str, attempt: int, max_attempts: int)

The job being run: its name, id and attempt (1 on the first run).

job_id is what send_job returned: use it as the idempotency key of the job's writes. attempt == max_attempts is the last try before the dead letters (Q13, R3JE-1).

JobSettings dataclass

JobSettings(dedupe_secs: int = 86400, retry_base_secs: float = 5.0, retry_max_secs: float = 300.0, retry_for_secs: float = 3600.0, dead_retention_secs: int = 604800, concurrency: int = 4, process_concurrency: int = 32, max_payload_bytes: int = 1048576, max_returns: int = 5)

Job limits of one process (LLMQ_JOB_*; owner decisions Q13-Q17).

Retries wait retry_base_secs, doubling up to retry_max_secs, until about retry_for_secs have passed; then the job is dead and kept dead_retention_secs.

max_attempts property

max_attempts: int

Tries before the dead letters: enough for the delays to cover retry_for_secs.

from_env classmethod

from_env(env: Mapping[str, str]) -> JobSettings

Read LLMQ_JOB_<FIELD> (for example LLMQ_JOB_DEDUPE_SECS); unset keeps the default.

delay_secs

delay_secs(attempt: int) -> float

The wait before the next try after failed attempt attempt (1-based).

EmfSink

EmfSink(write: Callable[[str], Any] | None = None, namespace: str = NAMESPACE, static: Mapping[str, str] | None = None, *, flush_secs: float = 1.0, properties: Mapping[str, str] | None = None)

CloudWatch embedded metric format, added up per label set and written once a second.

One line per point cost 17.7 us and 1,322 bytes of stdout per call (4 points; R3PF-14), and the alert tick wrote one line per gauge (R3OB-11). Now counters are summed, gauges keep their last value and samples go into a value array, per label set; each label set is one line with all its metrics, written when flush_secs has passed (checked on each point, and by the default writer's thread when idle), at exit, or by flush(). flush_secs=0 writes one line per point. properties (pod, version) ride on every line outside Dimensions: searchable in Logs Insights at no metric cost (R3OB-12).

write receives each line (default: stdout, via a writer thread); static labels too.

A write passed in runs on the event loop for every line, so it must not block. The default queues each line for a writer thread (EN-20: a backed-up stdout held the loop).

flush

flush() -> None

Write every pending point now.

count

count(name: str, value: float = 1.0, **labels: str) -> None

Add to a counter.

observe

observe(name: str, value: float, **labels: str) -> None

A sample (seconds for _seconds names).

gauge

gauge(name: str, value: float, **labels: str) -> None

A gauge point.

remove

remove(name: str, **labels: str) -> None

Nothing to remove: EMF lines are points, not state.

Metrics

Bases: Protocol

What the engine needs from a metrics sink.

count

count(name: str, value: float = 1.0, **labels: str) -> None

Add to a counter.

observe

observe(name: str, value: float, **labels: str) -> None

Record one sample of a distribution (seconds, tokens).

gauge

gauge(name: str, value: float, **labels: str) -> None

Set a gauge.

NullSink

Drops every point (tests, or metrics turned off).

count

count(name: str, value: float = 1.0, **labels: str) -> None

Ignore.

observe

observe(name: str, value: float, **labels: str) -> None

Ignore.

gauge

gauge(name: str, value: float, **labels: str) -> None

Ignore.

remove

remove(name: str, **labels: str) -> None

Ignore.

PrometheusSink

PrometheusSink(registry: Any = None)

prometheus_client metrics (needs the metrics extra); serve them with its exposition.

Create metrics lazily, by name and label set.

count

count(name: str, value: float = 1.0, **labels: str) -> None

Add to a counter (a point whose labels do not fit is logged and dropped).

observe

observe(name: str, value: float, **labels: str) -> None

Observe into a histogram (a point whose labels do not fit is logged and dropped).

gauge

gauge(name: str, value: float, **labels: str) -> None

Set a gauge (a point whose labels do not fit is logged and dropped).

remove

remove(name: str, **labels: str) -> None

Remove one gauge series, so a scrape stops showing a value nobody sets (R3OB-1).

ResultOffload

Bases: Protocol

Somewhere to keep a large result body.

put async

put(key: str, data: bytes) -> str

Store data; return a reference.

get async

get(ref: str) -> bytes | None

The data of a reference (None when it expired).

S3Offload

S3Offload(bucket: str, *, prefix: str = 'llmq-results/', key_prefix: str = '', session: AioSession | None = None, client_kwargs: dict[str, Any] | None = None, sse: str | None = 'AES256', kms_key_id: str | None = None, bucket_key: bool = False, expected_bucket_owner: str | None = None, delete_after_read: bool = True)

ResultOffload on an S3 bucket (server-side encryption; expiry by the bucket's rule).

Object names are {prefix}/{key_prefix}/{random id}: no caller text and no request id in a key (a hash of a guessable id told who made large results: R3JE-33), and no two writes on one name. A reference outside that prefix is refused before any read. sse=None leaves the header out (the bucket's default encryption applies); sse="aws:kms" with kms_key_id uses a customer key and bucket_key=True turns on S3 Bucket Keys (fewer KMS requests). expected_bucket_owner makes every request fail when the bucket changed owner. The client gets short timeouts (CLIENT_CONFIG) unless client_kwargs has a config. One client per event loop (never shared).

Keep settings; a client is created on first use on each loop.

put async

put(key: str, data: bytes) -> str

Write the object under a new random name; returns that name (key is not used).

delete async

delete(ref: str) -> None

Delete one object of this prefix (one no stored marker points to).

check_lifecycle async

check_lifecycle() -> bool

True when an enabled bucket rule expires objects under this prefix; else a warning.

Objects nobody reads (a caller that timed out, a delete that failed) are removed only by that rule (Q23). Needs s3:GetLifecycleConfiguration; access denied skips quietly.

get async

get(ref: str) -> bytes | None

Read the object (None when it is gone), then delete it; refuses a foreign reference.

aclose async

aclose() -> None

Close this loop's client (each loop closes its own); safe to repeat.

EmbeddingCodec dataclass

EmbeddingCodec(name: str, build_request: BuildRequest, parse_reply: ParseReply, max_batch: int, max_chars: int | None = None, patterns: tuple[str, ...] = (), input_types: frozenset[str] | None = None, probe_options: Mapping[str, Any] = (lambda: MappingProxyType({}))(), inference_names: frozenset[str] | None = None)

One embedding family: its limits, its request body and how its reply is read.

build_request(texts, options) returns the model's JSON body (Bedrock InvokeModel's body, or the OpenAI embeddings.create arguments); options holds the per-call input_type when one is given (registry inference defaults are merged later by the worker). parse_reply(body) gets the decoded reply body and returns one vector per text, in order; it raises ValueError / TypeError / KeyError on a body it cannot read. patterns are case-insensitive substrings of a model id that name this family. input_types, when set, is the closed set of input_type values the family takes (others fail before any call). probe_options go into the health probe's one-text request. inference_names are the registry inference names (dimensions, normalize, input_type, truncate, embedding_types) an InvokeModel body of this family takes: the worker fills those from the registry and drops the others; None leaves the body as built.

matches

matches(model_id: str) -> bool

True when one of patterns is in the model id, whatever its case (R3JE-21).

RegistryError

RegistryError(problems: list[str], warnings: list[str] | None = None)

Bases: ConfigError

The registry document is invalid; problems lists every problem with its path.

Keep the full lists; the message shows them one per line.

LiveEditRefused

Bases: ConfigError

A live edit touched fields that change only by deploy (or live edits are off).

PublishResult dataclass

PublishResult(action: str, version: int, carried: tuple[str, ...] = ())

What a publish did: published, kept, not_bumped or ignored, and the version live.

carried lists the live routing switches kept from the stored document (blocked regions and accounts, the kill switch), e.g. after a rollback (Q41).

RegistryConflict

Bases: ConfigError

A publish or live edit lost the race with other writes on every try it had.

The number of tries is Tuning.registry.edit_retries.

RegistryStore

RegistryStore(client: Redis, prefix: str, *, config: Mapping[str, Any] | None = None, live_edit: str = 'safe', allowed_hosts: tuple[str, ...] = (), audit_retention_days: int = AUDIT_RETENTION_DAYS, history_size: int = 20, edit_retries: int = 3, sleep: Callable[[float], Awaitable[None]] = asyncio.sleep)

The registry of one environment (key prefix) in the state store.

current is the last good ResolvedRegistry; hot paths read it without I/O. config is this process's own document, used by boot() and to restore a lost registry. For metrics: invalid_count counts refused reads, stored_refused is True while the stored document is refused (invalid or needing a newer library), lib_too_old while it needs a newer library (this pod never publishes over it), on_own_config while the pod runs on its own document.

Keep the client and settings; nothing is read or written until boot or load.

doc_key property

doc_key: str

The stored document (JSON string).

meta_key property

meta_key: str

Version record: version, config_hash, changes, published.

audit_key property

audit_key: str

Audit stream: one entry per publish or edit, path diffs with secrets redacted.

history_key property

history_key: str

Published versions kept for rollback (hash version -> document).

current property

current: ResolvedRegistry

Last good registry; raises ConfigError before the first successful load.

changes property

changes: int

Change counter of the loaded registry (-1 before the first load).

numbers_key

numbers_key(alias: str) -> str

A model's resolved numbers and the change counter they belong to.

live_edit_mode

live_edit_mode(doc: Mapping[str, Any]) -> str

The live-edit mode of a stored document: its live_edit, else the process setting.

load async

load() -> ResolvedRegistry | None

Read the stored registry; keep the last good one when it is unreadable or refused.

Returns None when there is no stored registry (nothing published yet, or it was lost). Concurrent calls share one read: every call that saw stale_config used to reload.

publish async

publish(doc: Mapping[str, Any], *, actor: str) -> PublishResult

Publish a document by the version rule; raise RegistryError when it is invalid.

The stored document's routing blocks and kill switch are carried over (R5). A stored document this library cannot read is treated as missing when its version is not above doc's (R1); one that needs a newer library is never replaced (lib_too_old, C5).

run_on_config

run_on_config() -> ResolvedRegistry

Run on this process's own document (the stored one is unreachable or refused).

Also the engine's fallback when boot runs out of time at start (a blackholed store).

boot async

boot(*, actor: str | None = None) -> ResolvedRegistry

Publish this process's config by the version rule, then load what is live.

When the store cannot be reached, or the live document is one this pod refuses (invalid and newer, or needing a newer library), the process starts on its own config (plan ยง3.18, R1). The audit actor defaults to boot@host:pid.

edit async

edit(change: Callable[[dict[str, Any]], None], *, actor: str) -> ResolvedRegistry

Apply change to the stored document as a live edit: validated, checked, audited.

Compare-and-set on the change counter; on a race the document is read again and the change re-applied, up to Tuning.registry.edit_retries tries with a jittered backoff between them; RegistryConflict when every try lost.

set async

set(values: Mapping[str, Any], *, actor: str, model: str | None = None) -> ResolvedRegistry

Set dotted paths ({"rpm": 400} with model=, or from the root); None deletes.

set_enabled async

set_enabled(model: str, deployment: str, enabled: bool, *, actor: str) -> ResolvedRegistry

Enable or disable one deployment (account/name) of a model.

block async

block(*, region: str | None = None, account: str | None = None, on: bool, actor: str) -> ResolvedRegistry

Block (or unblock) a region or an account for every model.

ConfigError when neither is given, or when blocking a name the registry does not use (a typo would otherwise block nothing and look done).

log async

log(count: int = 20, path_prefix: str = '') -> list[dict[str, Any]]

Latest audit entries, newest first; path_prefix keeps entries that touch it.

Best effort: it looks at the newest count * 5 entries, so a narrow path_prefix can return fewer than count; unreadable entries are skipped (logged).

versions async

versions() -> list[int]

Published versions still kept for rollback, oldest first.

rollback async

rollback(version: int, *, actor: str) -> PublishResult

Publish the content of an earlier version as a new, higher version.

The new version is the stored one plus one; when a deploy publishes in between (the rollback comes back ignored or not_bumped), it is read and tried again, up to Tuning.registry.edit_retries tries, then RegistryConflict. Each try is a whole publish, which itself retries edit_retries + 2 times on a lost race: at most edit_retries * (edit_retries + 2) publishes in all (3 x 5 by default; 10 x 12, about 135 s of backoff, at the maximum), not the 12.5 s bound of one publish (FC-16).

The live routing blocks and kill switch are kept, as for any publish, and the result's carried lists them (Q41): a rollback never lifts an emergency block.

diff_with_config async

diff_with_config() -> list[dict[str, Any]]

What is live but not in this process's config (live edits a deploy would drop).

refresh async

refresh() -> ResolvedRegistry | None

One reload: load, and restore the config when the stored registry was lost.

A missing or older stored registry while this process holds a newer one means the store was flushed or failed over: republish by the boot rule and alert (live edits are lost). A stored registry this pod refuses as invalid counts as missing when the pod's own version is not below it (R1); one that needs a newer library is left alone (C5).

run async

run(interval_secs: float) -> None

Reload every interval_secs until cancelled; errors never end the loop.

After a failed reload the wait doubles, up to 30 s (or the interval when longer), and goes back to the interval after a good one.

current_context

current_context() -> RequestContext

The active context, or the default (normal priority, no ids).

request_context

request_context(*, priority: bool | None = None, session_id: str | None = None, user_id: str | None = None, tenant_id: str | None = None) -> Iterator[RequestContext]

Set priority and ids for everything run inside the block; unset fields are inherited.

register_admission

register_admission(name: str, fn: AdmissionHook) -> None

admission: <name>: fn(ctx) -> Reject(reason) to refuse the call, None to admit it.

A hook that raises, or a name the registry uses that is not registered in this process, refuses the call (CallFailed("rejected")) and counts llmq_hook_failed_total (FO-4).

register_balancer

register_balancer(name: str, fn: Balancer) -> None

balance: "custom:<name>": fn(candidates, ctx) -> deployment keys, most preferred first.

register_fallback_policy

register_fallback_policy(name: str, fn: FallbackPolicy) -> None

fallback_policy: <name>: fn(cause, waited_secs, rule, ctx) -> may this call spill now.

register_probe

register_probe(name: str, fn: Probe) -> None

probe: <name>: fn(alias, op) -> the params a health probe of that model sends.

op is the provider operation (converse, chat, embeddings, invoke_model). Return None (or raise) to send the built-in probe instead. Called on the event loop: no blocking.

register_sticky_key

register_sticky_key(name: str, fn: StickyKey) -> None

sticky.key: "custom:<name>": fn(ctx) -> the key a call sticks by (None: no sticking).

run

run(main: Coroutine[Any, Any, T], *, use_uvloop: bool | None = None) -> T

Run main like asyncio.run, on uvloop when it is installed (extra uvloop, not Windows).

use_uvloop=None uses it when importable; True raises MissingExtra when it is not; False never does. uvloop speeds up the event loop itself (sockets, timers), not the SDKs.

register_provider

register_provider(name: str, factory: ProviderFactory) -> None

Add a provider kind (pair it with registry.register_account_kind).

current_job

current_job() -> JobContext | None

The job this code runs in, or None outside a job.

register_token_estimator

register_token_estimator(name: str, fn: Estimator) -> None

Add an estimator chosen by the token_estimator setting: fn(op, params) -> input tokens.

The setting is per model (or deployment), so one model can use a real tokenizer (e.g. for Indic or CJK text) while the others keep characters / 4.

register_embedding_family

register_embedding_family(name: str, codec: EmbeddingCodec) -> None

Add (or replace) an embedding family, so a registry model can name it in family.

Register before the registry is loaded, in every process that loads it (callers and workers). Patterns of the newest registration are tried first, so a narrower pattern (cohere.embed-v4) wins over a built-in one (cohere.embed).

register_account_kind

register_account_kind(kind: str, provider: str) -> None

Allow accounts of kind to serve models of provider (for a custom provider).