Operate¶
How to deploy, watch and change a running system.
Start and stop with the app¶
Start the queue once per process in the app's startup hook and stop it on shutdown:
from contextlib import asynccontextmanager
@asynccontextmanager
async def lifespan(app):
await llmq.start() # a failure here should fail the pod: queued models need it
yield
llmq.pause_receiving() # not ready; no new queued work
await llmq.stop() # drain: LLMQ_DRAIN_SECS (25 s) for jobs and calls together
Start one queue per process, after any fork (in each gunicorn / uvicorn worker, in Celery's
worker_process_init); see Clients and connections.
Deploys and restarts¶
- A preStop hook calls
llmq.pause_receiving()through a/drainendpoint of the app (the library's router has none), as insamples/09_http_router.py. It must answer GET:preStop: httpGetsends a GET, and a POST-only route answers 405 and nothing drains. Readiness fails, and the process takes no new calls or jobs; running work goes on. - Shutdown is
await llmq.stop(drain_secs=...). Held messages go back to the queue at once; running calls and this pod's waiting callers get up todrain_secs(defaultLLMQ_DRAIN_SECS, 25 s) to finish. Jobs drain first, then calls get what is left. - When the drain runs out, a call still running is given back, not failed: its grants are
refunded and its message returned to the queue at once, so another pod runs it and no failure
is counted toward
max_receives. A stream that already sent chunks to its caller cannot move and endsworker_shutdown(withpartial_text). A job still running is returned the same way.
sequenceDiagram
participant K as Kubernetes
participant P as Pod (app + llm-queue)
participant Q as Queue
K->>P: preStop: GET /drain
P->>P: pause_receiving(): not ready, receive nothing
K->>P: SIGTERM
P->>Q: return held (not started) messages
P->>P: running calls finish (up to 25 s)
P->>Q: still running: refund grants, return message
Q-->>K: another pod takes them
Note over K,P: terminationGracePeriodSeconds 30 > drain 25 s
Kubernetes settings that fit the defaults:
spec:
terminationGracePeriodSeconds: 30 # the default; keep it above LLMQ_DRAIN_SECS (25)
containers:
- name: app
lifecycle:
preStop:
httpGet: {path: /drain, port: 8000} # calls llmq.pause_receiving()
readinessProbe:
httpGet: {path: /llmq/health/ready, port: 8000}
livenessProbe:
httpGet: {path: /llmq/health/live, port: 8000}
The preStop hook's time counts against the grace period, so keep the /drain handler instant
(it only flips a flag). If you raise LLMQ_DRAIN_SECS for long answers, raise
terminationGracePeriodSeconds with it (drain + 5 s or more); a pod killed before its drain ends
loses nothing either, its calls reappear after visibility_secs, but they then count one
failure each.
Measured: 4 rolling restarts, 0 calls lost (gate B33).
Health and readiness¶
| Route / call | Answers |
|---|---|
GET /health/live |
200 while the library's background loops run and the heartbeat makes progress; 503 when a loop died or the heartbeat stalled for a lease. Once stop() begins it stays 200, so a draining pod is not restarted. It reads the process's own state only, never the store |
GET /health/ready |
200 when the registry is loaded and the process is not draining, else 503 |
await llmq.health() |
{"live", "ready", "stores": {"state", "queue"}, "eviction_policy"}, plus queue_eviction_policy with a separate queue store |
A liveness probe on /health/live restarts a pod whose loops died (0.1.0b1 always answered
200): keep its failureThreshold at 3 or more so one slow beat does not restart a healthy pod.
Readiness never depends on the stores' reachability or the providers, and answers at once from
cached state (one exception: in strict mode a store whose maxmemory-policy may evict keys makes
the process not ready, with not_ready_reason in the body; the policy is read again every
minute):
a background task probes each store every 5 s with a short write bounded by
EngineTimings.health_probe_secs (3 s) and reports it ok, read_only (a failover left it a
replica) or down (unreachable, or refusing writes, such as out of memory). A store error never
turns the readiness route into a 500. Liveness is false when one of the library's background
loops died or the heartbeat made no progress for a lease (LLMQ_LEASE_SECS).
At start() the store's own setup is checked: a cluster-mode store is refused (ConfigError),
and a maxmemory-policy other than noeviction logs store_eviction_policy with a warning and,
when strict, fails readiness (see
Store requirements).
Status¶
llm-queue status # every model
llm-queue status --model M # one model: deployments, budgets used, health, queues, blocks
llm-queue request ID # one call
The same from code: await llmq.status(model), await llmq.request_status(request_id); over
HTTP: GET /status, GET /requests/{id}.
Alerts¶
Every 30 s one process (the holder of the alert lock, which it keeps while it runs) evaluates
alerts from the status report. Each alert is a gauge llmq_alert_<name>, and a log line
llmq_alert is written once when it turns on or off; an alert whose model or deployment left
the registry is logged off. Thresholds come from the model's alerts setting. Alarm and query
examples: Monitor.
| Alert | On when |
|---|---|
model_unavailable |
No usable deployment is left: disabled, unhealthy, outside residency, blocked, on a disabled account, or behind an open breaker |
deployment_disabled |
A deployment was auto-disabled by health checks |
quota_near_limit |
A deployment used alerts.quota_used_pct (90) % of its RPM or TPM (off after two clear samples) |
quota_above_observed |
The provider throttled (this or the last minute) while the deployment had budget left: the real quota is below the registry's number (off after two clear samples) |
dead_letters |
A model's dead-letter queues (priority and normal together) hold alerts.dlq_messages (1) or more messages: dlq list --model M and dlq list --model M --priority |
breaker_open |
A deployment's breaker is open |
kill_switch_on |
The routing kill switch is on (one per model) |
queue_stalled |
A model's oldest queued call is older than alerts.queue_age_secs (600) |
job_dead_letters |
A job name has dead jobs |
Change things at run time¶
llm-queue registry set --model M rpm=400 # live edit (audited)
llm-queue registry disable M aws-main/ap-south-1
llm-queue block --region R # stop using a region (or --account A)
llm-queue unblock --region R
llm-queue drain M --yes --wait # disable a model for new calls; serve waiting ones
llm-queue dlq list --model M # dead letters; also show, redrive, purge
Commands that delete or move data (drain, dlq redrive, dlq purge, registry gc) need
--yes; --dry-run shows what they would do.
--actor NAME records who made a change. Live edits and their limits are in
Configure. Every command is in CLI and HTTP.
HTTP router¶
With the http extra:
from fastapi import Depends
app.include_router(
llmq.http_router(dependencies=[Depends(require_admin)]),
prefix="/llmq",
)
This gives /status, /requests/{id} and /registry behind your dependencies, and
/health/live and /health/ready without them. allow_writes=True, actor=fn adds
POST /registry and POST /registry/block, audited under actor(request). Write routes need
dependencies (else ConfigError), a JSON body and the X-LLMQ-Admin header; see
Secure a deployment.
Store failover¶
The store is one primary with replicas (cluster mode is not supported).
- Clients reconnect on
READONLYafter a failover. The failover drill (scripts/failover.py) loses no call onfailoverandrestart; with DNS lagging 6 s, at most 5 % of calls in flight may fail. LLMQ_QUEUE_WAIT_REPLICAS/LLMQ_QUEUE_WAIT_MS: a send waits for this many replicas to acknowledge it. Too few acknowledgements make the call run direct under the same record.- With a separate queue store (
LLMQ_QUEUE_STORE_URL), errors forLLMQ_QUEUE_FAILOVER_SECS(5) make calls run direct under the same budgets. Each send try is bounded by that time (a store that never answers fails over then, not after the client's ~21 s budget), and the process remembers the outage foroutage_memo_secs(3 s, cleared by a good store probe): calls in that time skip the queue store and run direct at once. - A state store outage: callers already waiting keep waiting until their deadline; new calls
follow the model's
store_downrule (pause, the default;static_cap;fail_fast). Measured: a 45 s outage with 500 calls waiting dropped none.
Details: Failures and recovery.
StoreUnavailable¶
Admin operations (cancel, status, request_status, send_job, and the registry's reads and
edits) raise StoreUnavailable when the store is unreachable, read-only or full. It is also a
ConnectionError. Calls raise CallFailed (queue_unavailable) instead.
Logs and metrics¶
The library logs with structlog and never configures logging; the app does. Log records are named events, for example:
| Event | Meaning |
|---|---|
llmq_call_finished |
One per call (every path, failures too): request id, model, outcome, account, region, deployment, attempts, waits, duration, tokens, worker, hashed user and tenant; no prompt text |
llmq_alert |
An alert turned on or off |
llmq_registry_audit |
A registry publish or edit, with actor and diff (secrets redacted) |
registry_older_ignored |
This process's registry is older than the stored one, which is used |
registry_invalid_replaced, registry_needs_newer_library |
The stored registry was invalid and replaced; or it needs a newer library and this pod keeps its own |
second_llm_queue_started |
A second LLMQueue was started in one process |
store_eviction_policy |
The store may evict keys (not noeviction) |
job_no_consumer |
A job was sent to a name no running process consumes |
store_down |
The state store failed before a call was sent; the store_down rule applies |
deployment_disabled, deployment_recovered |
Health checks turned a deployment off or back on |
Metrics go to LLMQ_METRICS_SINK: emf (CloudWatch embedded metrics, default, with the
environment dimension), prometheus (needs the metrics extra; status_router(...,
metrics_route=True) serves GET /metrics) or none; one sink per process. Per call: calls by
outcome and by caller-seen failure, attempts by result, tokens, and histograms of the queue
wait, the wait for a grant, the call and each provider attempt. Every 30 s the alert-lock
holder sets the cluster gauges (queues, held calls, oldest call, budgets and their limits,
concurrency, breakers, usable deployments, routing switches, dead jobs, alerts), and every pod
sets its own (ready, stores up, held calls, heartbeat age, outage memo, registry state, build
info). Monitor lists every metric with its labels, unit and emitter.