API reference¶
Generated from the docstrings of the public names in llm_queue.__all__.
llm_queue ¶
llm-queue: LLM calls through a queue, balanced across accounts and regions.
This is the public front door. LLMQueue (loaded on first use, so importing the package has
no side effects) runs calls through each model's queue under RPM / TPM budgets per deployment
(model x account x region), with fallback regions and models, health and status. The registry
(LLMQ_REGISTRY) describes accounts, models and settings; the extension points are plain
functions registered by name.
RequestContext
dataclass
¶
RequestContext(priority: bool = False, session_id: str | None = None, user_id: str | None = None, tenant_id: str | None = None)
What the current request is: its class and the ids used for sticky sessions and records.
CallFailed ¶
CallFailed(kind: Kind | str, message: str = '', *, retry_after_secs: float | None = None, partial_text: str | None = None, request_id: str | None = None)
Bases: LLMQueueError
A call ended without a reply; kind is one of KINDS.
Keep the kind and its details.
CallContext
dataclass
¶
CallContext(model: str, request_id: str, priority: bool, session_id: str | None, user_id: str | None, tenant_id: str | None, waited_secs: float = 0.0)
What hooks see of a call (no prompt text).
DeploymentView
dataclass
¶
DeploymentView(key: str, account: str, region: str, group: str, rpm: int | None, tpm: int | None, weight: float, tags: Mapping[str, Any] = dict())
What a custom balancer sees of one candidate deployment.
Reject
dataclass
¶
Returned by an admission hook to refuse a call before it is sent.
ConfigError ¶
Bases: LLMQueueError
Invalid configuration, raised at startup or on a refused edit (never mid-call).
MissingExtra ¶
Bases: ConfigError, ImportError
An optional feature's dependencies are not installed; names the extra to install.
Also an ImportError, so except ImportError around optional imports keeps working.
feature needs pip install 'llm-queue[extra]'.
StoreUnavailable ¶
Bases: LLMQueueError, ConnectionError
The state store could not serve an admin operation (unreachable, read-only, full).
Raised by cancel, status, send_job and the registry's edits and reads; a call
raises CallFailed instead. Also a ConnectionError, so existing handlers keep working.
UnknownModel ¶
Bases: ConfigError, KeyError
A model name that is neither an alias nor a model id in the registry.
Also a KeyError, so code that looks models up by key keeps working (HTTP 404, CLI).
BedrockOptions
dataclass
¶
BedrockOptions(read_timeout_secs: float = 300.0, connect_timeout_secs: float = 10.0, max_pool_connections: int = 100, role_session_secs: int = 3600, transport: str = 'aiohttp')
The Bedrock and STS clients (clients/pool.py).
EngineTimings
dataclass
¶
EngineTimings(idle_poll_secs: float = 0.05, idle_slow_secs: float = 1.0, idle_after_empty: int = 20, result_recheck_secs: float = 60.0, health_probe_secs: float = 3.0, stream_read_block_ms: int = 100, admission_probe_wait_ms: int = 200, admission_slot_wait_ms: int = 50, admission_slot_jitter_ms: int = 150, admission_wait_ms: int = 100, limiter_cut_ms: int = 1000, limiter_cut_factor: float = 0.7, budget_key_ms: int = 180000, throttle_base_ms: int = 1000, throttle_max_ms: int = 30000, slow_stale_ms: int = 300000, job_ttl_ms: int = 3600000)
Pacing of the engine's loops (engine/runtime.py, engine/streaming.py).
OpenAIOptions
dataclass
¶
OpenAIOptions(read_timeout_secs: float = 300.0, connect_timeout_secs: float = 10.0, max_connections: int = 100, max_keepalive_connections: int = 20, keepalive_secs: float = 15.0, http: Literal['httpx', 'aiohttp'] = 'aiohttp')
The OpenAI-compatible clients' timeouts, connection pool and HTTP client (one per account).
http="aiohttp" sends through the SDK's aiohttp client with the same limits and timeouts;
aiohttp has no idle cap, so max_keepalive_connections is not used there. Without one
(openai older than 1.9x, or no httpx-aiohttp / aiohttp), httpx is used and a warning logged.
RegistryOptions
dataclass
¶
The registry store (registry/store.py).
Audit retention stays LLMQ_AUDIT_RETENTION_DAYS.
StoreOptions
dataclass
¶
StoreOptions(retries: int = 3, connect_timeout_secs: float = 5.0, read_timeout_secs: float = 15.0, health_check_secs: float = 30.0, pool_wait_secs: float = 10.0)
The Valkey / Redis client of every process (engine/store_client.py).
The pool size stays LLMQ_STORE_MAX_CONNECTIONS: it follows the deployment's slots.
Tuning
dataclass
¶
Tuning(store: StoreOptions = StoreOptions(), bedrock: BedrockOptions = BedrockOptions(), engine: EngineTimings = EngineTimings(), registry: RegistryOptions = RegistryOptions(), openai: OpenAIOptions = OpenAIOptions(), warm: WarmOptions = WarmOptions())
All of the above; Tuning() is the defaults. Passed to LLMQueue.from_env(tuning=).
problems ¶
Every bad value and bad combination, one line each (empty when all is well).
queue_wait_ms is LLMQ_QUEUE_WAIT_MS: WAIT blocks a connection that long, so the
store read timeout must be above it. result_ttl_secs is LLMQ_RESULT_TTL_SECS: a
result must outlive a waiting caller's last-resort read (R3CF-10).
WarmOptions
dataclass
¶
Warm connections per Bedrock client (E4); pass connections=0 to turn it off.
After a client is built, connections unsigned HEAD requests go to its endpoint at once
through the client's own aiohttp session, so DNS, TCP and TLS are done before the first
call. They are not billable (no model is invoked; AWS answers an unsigned request with an
error status and keeps the connection). Every refresh_secs the same requests run again,
so the connections outlive AWS's ~20 s idle close and aiohttp's keepalive_secs.
Route
dataclass
¶
Where one attempt goes: account, region to call, model id, and the deployment's name.
Provider ¶
Bases: Protocol
One provider kind: runs an operation on a route and classifies its failures.
call
async
¶
call(op: str, params: Mapping[str, Any], route: Route, timing: Timing, sink: Sink | None = None, *, converse_api: str = 'converse') -> Result
Run one attempt; chat operations pass each streamed event to sink.
converse_api is the model's setting: converse or stream (Bedrock only).
DeploymentStatus ¶
Bases: TypedDict
One deployment of a model: switches, budgets, concurrency, breaker and latency.
account_enabled and blocked (a routing block on its region or account) were returned
but not declared, so a type checker refused reading them (R3AP-3).
EmbedResult
dataclass
¶
EmbedResult(vectors: list[list[float]], meta: Meta, errors: dict[int, CallFailed] = dict(), batches: int = 0, request_ids: list[str] = list())
Vectors (one per text, in order) and how the batches were served.
With embed(partial=True), a text that failed has the vector [] and its error in
errors (by text index); otherwise errors is always empty. meta.tokens_in sums the
input tokens the replies reported; batches counts provider calls and request_ids
lists the answered ones, for cost and support (Q21).
Health ¶
Bases: TypedDict
Liveness, readiness, each store's state (ok, read_only, down) and eviction policy.
eviction_policy is the state store's maxmemory-policy read at start (unknown when
unreadable); queue_eviction_policy is the queue store's, when it is a separate store.
LLMQueue ¶
LLMQueue(engine: Engine, *, settings: InfraSettings | None = None, builder: Callable[[], Engine] | None = None, offload: ResultOffload | None = None, job_settings: JobSettings | None = None, tuning: Tuning | None = None)
One process's LLM queue: caller, worker, jobs and admin over the stores.
Wrap a built Engine; builder makes a caller-only engine for sync calls.
That engine is used only while this queue is not started (sync-only processes).
job_settings (default: LLMQ_JOB_* read by from_env) bounds jobs. tuning is
the one the engine was built with (from_env passes it): start() compares its
client timeouts with the registry (R3CF-10).
engine
property
¶
Deprecated: the engine is private (owner decision Q35); use the queue's methods.
Kept as an alias that warns until 0.2.
jobs
property
¶
Deprecated: the job runner is private (Q35); use register_job and send_job.
from_env
classmethod
¶
from_env(environ: Mapping[str, str] | None = None, *, secrets: Secrets | None = None, metrics: Metrics | None = None, worker_id: str | None = None, credentials: Callable[[Any], Mapping[str, Any] | None] | None = None, offload: ResultOffload | None = None, offload_bytes: int | None = None, tuning: Tuning | None = None) -> LLMQueue
Build from LLMQ_* (secret first when secrets is given, then the environment).
tuning sets timeouts, pool sizes and loop pacing in code (Tuning() is the
defaults); a bad value or combination raises ConfigError listing every problem.
start
async
¶
Publish / load the registry; run the worker side unless mode=caller.
A second start() does nothing; a stopped queue cannot start again (build a new one).
The first started queue of the process is the LangChain default and stays so. Starts
run one at a time, on one event loop. A start that fails part-way stops what it
started and re-raises; the queue is then stopped (a stopped engine cannot start
again), and a later start() raises LLMQueueError naming that first error.
pause_receiving ¶
Fail readiness and take no new calls or jobs; running work goes on (preStop hook).
stop
async
¶
Shut down in order (plan ยง3.18) and close every client; a second call does nothing.
drain_secs (default LLMQ_DRAIN_SECS) bounds the whole wait: jobs first (they make
calls), then running calls get what is left of it. Calls made during the drain are
served; new calls are refused once it is over. Every step runs even when an earlier
one fails (each failure is logged) and the first failure is raised at the end; if the
stop is cancelled, the steps left still close their clients, without draining.
call
async
¶
Any operation of a model (converse, chat, invoke_model, embeddings).
converse, chat and call run the same code: the model's provider, not the method
name, decides the request shape (Converse for Bedrock, chat completions for
OpenAI-compatible). Use call for code that handles every model alike.
Per-call options (keyword only; R3AP-10): timeout_secs (default the model's
timeout_secs, capped by timeout_max_secs), max_wait_secs (longest wait for
budget), request_id (your id; a repeat attaches to the same call), session_id,
user_id, tenant_id (default from request_context), allow_fallback ("all":
fallback regions and models, "regions": regions only, "none"), allowed_regions
and allowed_accounts (limit the deployments). An unknown option raises TypeError.
converse
async
¶
converse(model: str, request: Mapping[str, Any], *, priority: bool | None = None, **kw: Any) -> Reply
One chat call, named for Bedrock: send a Converse request to a Bedrock model.
Same options and code as call; the reply's response is the Converse body.
chat
async
¶
One chat call, named for OpenAI: send a chat completion to an OpenAI-compatible model.
Same options and code as call; the reply's response is the completion body.
converse_stream
async
¶
converse_stream(model: str, request: Mapping[str, Any], *, priority: bool | None = None, **kw: Any) -> AsyncIterator[dict[str, Any]]
Stream a chat call's events as they arrive (plan ยง3.13).
Bedrock models yield ConverseStream events, OpenAI-compatible ones chat completion
chunks. A failure after events were yielded raises CallFailed with partial_text.
chat_stream
async
¶
chat_stream(model: str, request: Mapping[str, Any], *, priority: bool | None = None, **kw: Any) -> AsyncIterator[dict[str, Any]]
converse_stream under the OpenAI name: chat completion chunks from such a model.
Was an alias, so help() and tracebacks showed converse_stream (R3AP-15).
embed
async
¶
embed(model: str, texts: Sequence[str], *, priority: bool | None = None, input_type: str | None = None, partial: bool = False, **kw: Any) -> EmbedResult
One vector per text, in order (batched by the model's embed_batch_size).
partial=True keeps the batches that succeeded (see EmbedResult.errors); by default
one failed batch fails the call. request_id=X names batch n X:n.
converse_sync ¶
converse_sync(model: str, request: Mapping[str, Any], *, priority: bool | None = None, **kw: Any) -> Reply
converse on the direct path from sync code (same budgets, no queue).
chat_sync ¶
chat_sync(model: str, request: Mapping[str, Any], *, priority: bool | None = None, **kw: Any) -> Reply
chat on the direct path from sync code.
embed_sync ¶
embed_sync(model: str, texts: Sequence[str], *, priority: bool | None = None, input_type: str | None = None, partial: bool = False, **kw: Any) -> EmbedResult
embed on the direct path from sync code.
close_sync ¶
Stop the private engine of the sync calls (sync-only processes; stop() does too).
register_job ¶
register_job(name: str, fn: JobFn, *, timeout_secs: float | None = None, concurrency: int | None = None) -> None
Allow this process to run job name (call before start).
concurrency caps its runs in this process (default LLMQ_JOB_CONCURRENCY, 4).
A run longer than timeout_secs is cancelled and counts as a failure (retried, then
dead-lettered); None (the default) lets it run as long as it takes. ConfigError in
caller mode, which runs no jobs: the job waited unseen (R3JE-6).
send_job
async
¶
Queue a job; returns its id (jobs run at least once).
A second send with the same job_id within LLMQ_JOB_DEDUPE_SECS (24 h) queues
nothing and returns the id. CallFailed("queue_full") when 10,000 jobs of this name
already wait; request_too_large above LLMQ_JOB_MAX_PAYLOAD_BYTES (1 MB);
invalid_request for a payload that is not JSON.
job_dead_letters
async
¶
Dead jobs of one name, oldest first: entry id, job id, failures, payload (Q15).
redrive_jobs
async
¶
Send up to count dead jobs of one name back to run again; returns how many.
status
async
¶
Status of a model (ModelStatus) or of everything (StatusReport).
request_status
async
¶
Status of one call (empty when unknown).
http_router ¶
http_router(*, dependencies: Sequence[Any] = (), allow_writes: bool = False, actor: Callable[[Any], str] | None = None) -> Any
A FastAPI router with status, requests, registry and health (extra http).
Writes need actor(request) -> str, the authenticated name recorded in the audit.
ModelStatus ¶
RequestStatus ¶
Bases: TypedDict
One call's status record as stored (empty when the id is unknown).
Every value is text, as the store keeps it: convert the numbers with int() (attempts,
tokens_in, tokens_out) and the *_at times (Unix seconds) with float(). One *_at
key is there per state the call reached (R3AP-4).
Meta ¶
Bases: BaseModel
How a call was served: deployment, fallback, attempts, timings, tokens.
Reply ¶
Bases: BaseModel
The stored result of a call: the provider reply, or an error kind with its details.
dumps ¶
JSON bytes for the store (fields directly: model_dump re-walked the response).
JobContext
dataclass
¶
The job being run: its name, id and attempt (1 on the first run).
job_id is what send_job returned: use it as the idempotency key of the job's writes.
attempt == max_attempts is the last try before the dead letters (Q13, R3JE-1).
JobSettings
dataclass
¶
JobSettings(dedupe_secs: int = 86400, retry_base_secs: float = 5.0, retry_max_secs: float = 300.0, retry_for_secs: float = 3600.0, dead_retention_secs: int = 604800, concurrency: int = 4, process_concurrency: int = 32, max_payload_bytes: int = 1048576, max_returns: int = 5)
Job limits of one process (LLMQ_JOB_*; owner decisions Q13-Q17).
Retries wait retry_base_secs, doubling up to retry_max_secs, until about
retry_for_secs have passed; then the job is dead and kept dead_retention_secs.
EmfSink ¶
EmfSink(write: Callable[[str], Any] | None = None, namespace: str = NAMESPACE, static: Mapping[str, str] | None = None, *, flush_secs: float = 1.0, properties: Mapping[str, str] | None = None)
CloudWatch embedded metric format, added up per label set and written once a second.
One line per point cost 17.7 us and 1,322 bytes of stdout per call (4 points; R3PF-14), and
the alert tick wrote one line per gauge (R3OB-11). Now counters are summed, gauges keep
their last value and samples go into a value array, per label set; each label set is one
line with all its metrics, written when flush_secs has passed (checked on each point, and
by the default writer's thread when idle), at exit, or by flush(). flush_secs=0 writes
one line per point. properties (pod, version) ride on every line outside Dimensions:
searchable in Logs Insights at no metric cost (R3OB-12).
write receives each line (default: stdout, via a writer thread); static labels too.
A write passed in runs on the event loop for every line, so it must not block. The
default queues each line for a writer thread (EN-20: a backed-up stdout held the loop).
NullSink ¶
Drops every point (tests, or metrics turned off).
PrometheusSink ¶
prometheus_client metrics (needs the metrics extra); serve them with its exposition.
Create metrics lazily, by name and label set.
count ¶
Add to a counter (a point whose labels do not fit is logged and dropped).
observe ¶
Observe into a histogram (a point whose labels do not fit is logged and dropped).
gauge ¶
Set a gauge (a point whose labels do not fit is logged and dropped).
remove ¶
Remove one gauge series, so a scrape stops showing a value nobody sets (R3OB-1).
ResultOffload ¶
S3Offload ¶
S3Offload(bucket: str, *, prefix: str = 'llmq-results/', key_prefix: str = '', session: AioSession | None = None, client_kwargs: dict[str, Any] | None = None, sse: str | None = 'AES256', kms_key_id: str | None = None, bucket_key: bool = False, expected_bucket_owner: str | None = None, delete_after_read: bool = True)
ResultOffload on an S3 bucket (server-side encryption; expiry by the bucket's rule).
Object names are {prefix}/{key_prefix}/{random id}: no caller text and no request id in
a key (a hash of a guessable id told who made large results: R3JE-33), and no two writes on
one name. A reference outside that prefix is refused before any read. sse=None leaves the
header out (the bucket's default encryption applies); sse="aws:kms" with kms_key_id
uses a customer key and bucket_key=True turns on S3 Bucket Keys (fewer KMS requests).
expected_bucket_owner makes every request fail when the bucket changed owner. The client
gets short timeouts (CLIENT_CONFIG) unless client_kwargs has a config. One client per
event loop (never shared).
Keep settings; a client is created on first use on each loop.
put
async
¶
Write the object under a new random name; returns that name (key is not used).
delete
async
¶
Delete one object of this prefix (one no stored marker points to).
check_lifecycle
async
¶
True when an enabled bucket rule expires objects under this prefix; else a warning.
Objects nobody reads (a caller that timed out, a delete that failed) are removed only
by that rule (Q23). Needs s3:GetLifecycleConfiguration; access denied skips quietly.
get
async
¶
Read the object (None when it is gone), then delete it; refuses a foreign reference.
aclose
async
¶
Close this loop's client (each loop closes its own); safe to repeat.
EmbeddingCodec
dataclass
¶
EmbeddingCodec(name: str, build_request: BuildRequest, parse_reply: ParseReply, max_batch: int, max_chars: int | None = None, patterns: tuple[str, ...] = (), input_types: frozenset[str] | None = None, probe_options: Mapping[str, Any] = (lambda: MappingProxyType({}))(), inference_names: frozenset[str] | None = None)
One embedding family: its limits, its request body and how its reply is read.
build_request(texts, options) returns the model's JSON body (Bedrock InvokeModel's body,
or the OpenAI embeddings.create arguments); options holds the per-call input_type when
one is given (registry inference defaults are merged later by the worker).
parse_reply(body) gets the decoded reply body and returns one vector per text, in order;
it raises ValueError / TypeError / KeyError on a body it cannot read.
patterns are case-insensitive substrings of a model id that name this family.
input_types, when set, is the closed set of input_type values the family takes (others
fail before any call). probe_options go into the health probe's one-text request.
inference_names are the registry inference names (dimensions, normalize,
input_type, truncate, embedding_types) an InvokeModel body of this family takes: the
worker fills those from the registry and drops the others; None leaves the body as built.
matches ¶
True when one of patterns is in the model id, whatever its case (R3JE-21).
RegistryError ¶
Bases: ConfigError
The registry document is invalid; problems lists every problem with its path.
Keep the full lists; the message shows them one per line.
LiveEditRefused ¶
Bases: ConfigError
A live edit touched fields that change only by deploy (or live edits are off).
PublishResult
dataclass
¶
What a publish did: published, kept, not_bumped or ignored, and the version live.
carried lists the live routing switches kept from the stored document (blocked regions
and accounts, the kill switch), e.g. after a rollback (Q41).
RegistryConflict ¶
Bases: ConfigError
A publish or live edit lost the race with other writes on every try it had.
The number of tries is Tuning.registry.edit_retries.
RegistryStore ¶
RegistryStore(client: Redis, prefix: str, *, config: Mapping[str, Any] | None = None, live_edit: str = 'safe', allowed_hosts: tuple[str, ...] = (), audit_retention_days: int = AUDIT_RETENTION_DAYS, history_size: int = 20, edit_retries: int = 3, sleep: Callable[[float], Awaitable[None]] = asyncio.sleep)
The registry of one environment (key prefix) in the state store.
current is the last good ResolvedRegistry; hot paths read it without I/O. config is
this process's own document, used by boot() and to restore a lost registry. For metrics:
invalid_count counts refused reads, stored_refused is True while the stored document is
refused (invalid or needing a newer library), lib_too_old while it needs a newer library
(this pod never publishes over it), on_own_config while the pod runs on its own document.
Keep the client and settings; nothing is read or written until boot or load.
audit_key
property
¶
Audit stream: one entry per publish or edit, path diffs with secrets redacted.
history_key
property
¶
Published versions kept for rollback (hash version -> document).
current
property
¶
Last good registry; raises ConfigError before the first successful load.
numbers_key ¶
A model's resolved numbers and the change counter they belong to.
live_edit_mode ¶
The live-edit mode of a stored document: its live_edit, else the process setting.
load
async
¶
Read the stored registry; keep the last good one when it is unreadable or refused.
Returns None when there is no stored registry (nothing published yet, or it was lost).
Concurrent calls share one read: every call that saw stale_config used to reload.
publish
async
¶
Publish a document by the version rule; raise RegistryError when it is invalid.
The stored document's routing blocks and kill switch are carried over (R5). A stored
document this library cannot read is treated as missing when its version is not above
doc's (R1); one that needs a newer library is never replaced (lib_too_old, C5).
run_on_config ¶
Run on this process's own document (the stored one is unreachable or refused).
Also the engine's fallback when boot runs out of time at start (a blackholed store).
boot
async
¶
Publish this process's config by the version rule, then load what is live.
When the store cannot be reached, or the live document is one this pod refuses (invalid
and newer, or needing a newer library), the process starts on its own config (plan
ยง3.18, R1). The audit actor defaults to boot@host:pid.
edit
async
¶
Apply change to the stored document as a live edit: validated, checked, audited.
Compare-and-set on the change counter; on a race the document is read again and the
change re-applied, up to Tuning.registry.edit_retries tries with a jittered backoff
between them; RegistryConflict when every try lost.
set
async
¶
Set dotted paths ({"rpm": 400} with model=, or from the root); None deletes.
set_enabled
async
¶
Enable or disable one deployment (account/name) of a model.
block
async
¶
block(*, region: str | None = None, account: str | None = None, on: bool, actor: str) -> ResolvedRegistry
Block (or unblock) a region or an account for every model.
ConfigError when neither is given, or when blocking a name the registry does not use
(a typo would otherwise block nothing and look done).
log
async
¶
Latest audit entries, newest first; path_prefix keeps entries that touch it.
Best effort: it looks at the newest count * 5 entries, so a narrow path_prefix can
return fewer than count; unreadable entries are skipped (logged).
rollback
async
¶
Publish the content of an earlier version as a new, higher version.
The new version is the stored one plus one; when a deploy publishes in between (the
rollback comes back ignored or not_bumped), it is read and tried again, up to
Tuning.registry.edit_retries tries, then RegistryConflict. Each try is a whole
publish, which itself retries edit_retries + 2 times on a lost race: at most
edit_retries * (edit_retries + 2) publishes in all (3 x 5 by default; 10 x 12, about
135 s of backoff, at the maximum), not the 12.5 s bound of one publish (FC-16).
The live routing blocks and kill switch are kept, as for any publish, and the result's
carried lists them (Q41): a rollback never lifts an emergency block.
diff_with_config
async
¶
What is live but not in this process's config (live edits a deploy would drop).
refresh
async
¶
One reload: load, and restore the config when the stored registry was lost.
A missing or older stored registry while this process holds a newer one means the store was flushed or failed over: republish by the boot rule and alert (live edits are lost). A stored registry this pod refuses as invalid counts as missing when the pod's own version is not below it (R1); one that needs a newer library is left alone (C5).
run
async
¶
Reload every interval_secs until cancelled; errors never end the loop.
After a failed reload the wait doubles, up to 30 s (or the interval when longer), and goes back to the interval after a good one.
current_context ¶
The active context, or the default (normal priority, no ids).
request_context ¶
request_context(*, priority: bool | None = None, session_id: str | None = None, user_id: str | None = None, tenant_id: str | None = None) -> Iterator[RequestContext]
Set priority and ids for everything run inside the block; unset fields are inherited.
register_admission ¶
admission: <name>: fn(ctx) -> Reject(reason) to refuse the call, None to admit it.
A hook that raises, or a name the registry uses that is not registered in this process,
refuses the call (CallFailed("rejected")) and counts llmq_hook_failed_total (FO-4).
register_balancer ¶
balance: "custom:<name>": fn(candidates, ctx) -> deployment keys, most preferred first.
register_fallback_policy ¶
fallback_policy: <name>: fn(cause, waited_secs, rule, ctx) -> may this call spill now.
register_probe ¶
probe: <name>: fn(alias, op) -> the params a health probe of that model sends.
op is the provider operation (converse, chat, embeddings, invoke_model). Return
None (or raise) to send the built-in probe instead. Called on the event loop: no blocking.
register_sticky_key ¶
sticky.key: "custom:<name>": fn(ctx) -> the key a call sticks by (None: no sticking).
run ¶
Run main like asyncio.run, on uvloop when it is installed (extra uvloop, not Windows).
use_uvloop=None uses it when importable; True raises MissingExtra when it is not;
False never does. uvloop speeds up the event loop itself (sockets, timers), not the SDKs.
register_provider ¶
Add a provider kind (pair it with registry.register_account_kind).
register_token_estimator ¶
Add an estimator chosen by the token_estimator setting: fn(op, params) -> input tokens.
The setting is per model (or deployment), so one model can use a real tokenizer (e.g. for Indic or CJK text) while the others keep characters / 4.
register_embedding_family ¶
Add (or replace) an embedding family, so a registry model can name it in family.
Register before the registry is loaded, in every process that loads it (callers and
workers). Patterns of the newest registration are tried first, so a narrower pattern
(cohere.embed-v4) wins over a built-in one (cohere.embed).
register_account_kind ¶
Allow accounts of kind to serve models of provider (for a custom provider).