Skip to content

Extension points

Named hooks an app registers to change one decision without forking the library.

Hook Registered with Chosen by Contract
Balancer register_balancer(name, fn) balance: "custom:<name>" fn(views, ctx) -> ordered keys; the script then takes the first that fits
Fallback policy register_fallback_policy(name, fn) fallback_policy fn(cause, waited, rule, ctx) -> bool replaces the on / after_secs test
Admission register_admission(name, fn) admission fn(ctx) returns a Reject or None, on the caller before sending
Sticky key register_sticky_key(name, fn) sticky.key: "custom:<name>" fn(ctx) returns a key or None (hashed)
Probe register_probe(name, fn) probe fn(alias, op) -> params or None (type Probe) for health probes; op is converse, chat, embeddings or invoke_model; None sends the built-in probe
Token estimator register_token_estimator(name, fn) token_estimator fn(op, params) -> input tokens
Embedding family register_embedding_family(name, codec) model family, or the codec's id patterns an EmbeddingCodec: build_request(texts, options), parse_reply(body), max_batch, max_chars; register before the registry loads (Embedding families)
Provider, account kind register_provider(name, factory), register_account_kind(kind, provider) model provider, account kind a Provider with call(op, params, route, timing, sink=None, *, converse_api="converse") and classify(exc)
Others register_job; from_env(credentials=, secrets=, metrics=, offload=, offload_bytes=, worker_id=) credentials per account, secret source, metrics sink, S3 offload

Hooks run through call_safely. A balancer, fallback policy, sticky key, probe or token estimator that raises or is not registered is logged once and the built-in behaviour is used, so those hooks never fail a call.

Admission fails closed. An admission hook is a gate (tenants, entitlements), so one that raises, or an admission name the registry uses that is not registered in this process, refuses the call with CallFailed("rejected"). Every hook failure, of any kind, is logged once per name and counted on every occurrence as llmq_hook_failed_total{kind, hook} (admission hooks; the others are logged only).

Hooks must not block. They run synchronously on the caller's or worker's event loop: no network or disk I/O and no time.sleep inside a hook, or every call in the process waits on it. Load what a hook needs (entitlements, prices) in the background and read it from memory.

CallContext carries ids, never prompt text.