Skip to content

CLI and HTTP

llm-queue command

Reads the same LLMQ_* settings as the app. --actor NAME (before the command) records who made a change in the audit; without it the OS user is used, looked up only by commands that write (uid:<n> when the user has no passwd entry, as in a container run under a bare UID).

Command Does
llm-queue status [--model M] Status of every model, or of one
llm-queue request ID Status of one call (exit 1 when unknown or expired)
llm-queue settings get --model M [--deployment D] Effective value of every setting and the level it came from
llm-queue registry get [--model M] The live registry (secrets redacted)
llm-queue registry set [--model M] path=value ... Live edit (JSON values)
llm-queue registry enable M D / disable M D Turn one deployment of a model on or off. D is account/name, where name is the region or the inference profile's name as listed under confirmed / fallback; the edit is written to deployments.<account>.overrides.<name>
llm-queue registry diff What is live but not in this process's config
llm-queue registry log [--count N] [--path P] The audit trail (default 20 entries): publishes, edits and CLI cancels
llm-queue registry publish [--from-version N] Publish LLMQ_REGISTRY, or roll back to version N
llm-queue registry gc --yes\|--dry-run Delete the keys and queues of models that no longer exist
llm-queue registry schema The registry document's JSON Schema (Registry document); needs no store and no LLMQ_* settings
llm-queue block --region R / --account A Stop using a region or an account
llm-queue unblock --region R / --account A Use it again
llm-queue cancel ID [ID ...] --yes\|--dry-run Cancel calls for every caller of each id (at most 1000 ids). Anyone with store access may cancel any id; each cancel is logged (call_cancel, source=cli) and written to the audit trail with the actor
llm-queue drain M --yes\|--dry-run [--wait] [--timeout S] Disable a model for new calls; serve waiting ones (--wait until its queues are empty, default timeout 600 s; exit 3 when it times out)
llm-queue dlq list\|show\|redrive\|purge --model M [--priority] [--count N] Dead letters of a model's queue (show --unsafe prints message bodies; redrive and purge need --yes or --dry-run)
llm-queue queues list The queues

Commands that delete or move data (dlq purge, dlq redrive, registry gc, drain, cancel) refuse to run without --yes; --dry-run shows what they would do. The CLI reads the store password like the app does. --actor is 1-200 characters without control characters.

drain stays until it is undone. It is a live edit (enabled: false on the model), so the model takes no new calls until someone sets it back: llm-queue registry set --model M enabled=true. It is refused when live edits are off (LLMQ_REGISTRY_LIVE_EDIT=off); then change enabled in the registry and publish.

Exit codes. 0 done; 1 an error (a bad name or edit, an unknown or expired request id, a store outage: one line on stderr, no traceback); 3 a drain --wait that ran out of time with calls still waiting.

HTTP routes

llmq.http_router(dependencies=[...], allow_writes=False, actor=None) returns a FastAPI router (extra http):

app.include_router(llmq.http_router(dependencies=[Depends(require_admin)]), prefix="/llmq")
Route Auth Answers
GET /health/live none 200 while the library's background loops run and the heartbeat makes progress; 503 when a loop died or the heartbeat stalled for a lease (stays 200 once stop() begins). Reads the process's own state, never the store
GET /health/ready none 200 when the registry is loaded and not draining, else 503; each store as ok, read_only or down (from a probe cached every 5 s; never a 500)
GET /status?model=M dependencies Status of every model or one
GET /requests/{id} dependencies Status of one call
GET /registry dependencies The live registry (role ARNs and secret references redacted)
POST /registry?model=M dependencies, allow_writes=True, header X-LLMQ-Admin Live edit: a JSON body of path: value (at most 100 paths), audited under actor(request)
POST /registry/block dependencies, allow_writes=True, header X-LLMQ-Admin Block or unblock: a JSON body {"region": R} or {"account": A}, with "on": true\|false (default true), audited

Status codes of the admin routes (every route except the two health routes):

Code When
200 Done
400 A refused write: a live edit of a protected or deploy-only field, an invalid value, a bad block body, more than 100 paths, an id or name longer than 256 characters
403 A write while allow_writes is off, or without the X-LLMQ-Admin header
404 An unknown model (/status), an unknown or expired request id (/requests/{id})
409 A write lost the compare-and-set race with other writes on every try; the request is valid, retry after Retry-After (1 s)
422 A write body that is not a JSON object (FastAPI's own validation)
503 The state store is unreachable, or the registry is not loaded yet (the router was hit before start() finished)

allow_writes=True needs dependencies= (else ConfigError) and actor= (a function from the request to the authenticated name). With no dependencies the read routes are open to anyone who reaches the app (/registry lists accounts, regions, model ids and base URLs), and the router logs admin_routes_unauthenticated when it is built. See Secure a deployment.

The /drain route used by a preStop hook is the app's own (see samples/09_http_router.py); the router has none. It must answer GET, which is what preStop: httpGet sends.