Skip to content

Failures and recovery

How the library behaves when a worker, the store, a provider or a caller fails. Every row is exercised by a fault scenario (scripts/scenarios.py) or a gate.

Circuit breaker

stateDiagram-v2
  direction LR
  [*] --> Closed
  Closed --> Closed: success / failure below threshold
  Closed --> Open: breaker_threshold failures<br/>within breaker_window_secs
  Open --> HalfOpen: breaker_cooldown_secs passed
  HalfOpen --> Closed: the one probe call succeeds
  HalfOpen --> Open: the probe call fails
  note right of HalfOpen
    One probe call at a time.
    Other calls wait for its answer
    (admission_probe_wait_ms),
    they do not fail.
  end note

State store outage

flowchart TD
  X([A store command fails]) --> Y{Where?}
  Y -- "caller, before the send" --> SD{store_down rule}
  SD -- pause --> SP[Ping every second,<br/>then run the call normally]
  SD -- static_cap --> SC[Run without the store,<br/>at most LLMQ_LOCAL_LIMIT_CAP per process]
  SD -- fail_fast --> SF([queue_unavailable])
  Y -- "caller, after the send" --> SW[Wait for the store<br/>never run the call twice]
  Y -- "worker" --> WT[Mark store trouble,<br/>retry; give grants back on errors]
  SP & SC & SW --> DL{Deadline}
  DL -- passed --> DE([deadline_exceeded or queue_unavailable])
  WT --> G[For a caller TTL after any trouble<br/>or a late heartbeat, callers count as alive]

Applies when the state store fails before the call was sent (registering). pause (default): ping every second until the call's timeout, then run it normally, else queue_unavailable. static_cap: run without the store, at most LLMQ_LOCAL_LIMIT_CAP (10) calls per process and model, confirmed deployments in order, no budgets (quotas may be exceeded). fail_fast: queue_unavailable at once. A call already sent never takes this path: its caller waits for the store until its deadline (before, it could run twice). A streaming call fails queue_unavailable (chunks travel through the store).

Failure model

Event What happens
Worker killed mid-call Its message reappears after visibility_secs (60) with one failure counted; its grant's lease lapses and SWEEP frees the slot and entry (the charged budget stays); PICK_TAKE drops a dead owner's entry when the message is admitted again. A result already stored is reused. After max_receives (10) failures the message is dead-lettered and the caller gets dead_lettered (B13 / B30 / B71 tested)
Error inside the worker (store refusal, bug) _release: every grant settled as a cut, message returned at once (or in 5 s for missing numbers); a direct call's caller gets the error. A store blip after the provider answered skips FINISH (the lease expires) but keeps the answer
State store outage Callers already waiting keep waiting for the store; new calls follow store_down, all within the call's one deadline (timeout + caller_margin_secs). A worker treats callers as alive for a caller TTL + a heartbeat + 5 s after it saw the store fail (any store error, a late heartbeat, or a heartbeat back from a gap), so waiting calls are not dropped as caller_gone (B95: 45 s block, 500 waiting, 0 dropped; scenarios S4, S5). The registry stays on its last good copy. Admin operations raise StoreUnavailable
Half-open store (replies never come) Every store command fails after Tuning.store.read_timeout_secs (15 s); a timeout is not retried (redis-py retries it by default: 4 x 15 s, and a Lua write could run twice). Scenario S5
Store out of memory Calls fail as queue_unavailable ("out of memory"), admin operations as StoreUnavailable (scenario S9)
A call still held past its deadline Should not happen; the heartbeat logs held_call_stuck, cancels its task, gives its grants back and returns its message (one-hour soak: two calls were renewed for 50 minutes)
Store failover Reconnect on READONLY; scripts/failover.py: failover and restart lose no call; slowdns (DNS lags 6 s) may fail calls in flight, at most 5 %
Queue store outage (separate store) Per call: send retries for 5 s, then the call runs direct under the same budgets
Provider throttle Refund all, limiter cut; the deployment is out for every pod until max(Retry-After, a jittered backoff) passes, and the call moves to another deployment at once or waits for the window when none is left. Throttles count in the retry budget (Throttling); alert quota_above_observed when throttles come with budget left
Provider errors Breaker per deployment; retries capped by attempts, a per-model retry budget and the deadline. A move to a deployment the call has not tried is free of the retry budget (failover, bounded by the deployments); retries after every deployment failed spend it (scenarios P3, P4, P7)
Credentials refused The account is skipped in that region only, with backoff (5 s doubling to 60 s, reset by the next success): an opt-in region such as ap-south-2 can refuse a token the others accept (scenario P5). Signature and clock-skew errors count as credential errors. Assumed-role sessions and credentials= values with an expiry are refreshed about 5 minutes before they expire
Connection reset by the provider One quick retry on the same deployment, with no breaker or retry-budget charge; a second reset is a normal failure
Deployment broken (target errors) disable_errors (3) distinct requests in disable_window_secs (300) with no success between disable it, unless it is the model's last healthy one (then only a log); probes after probe_after_secs (900), then doubling (at least 60 s) up to probe_max_secs (7200); health is apart from the registry, so a deploy never re-enables one
No usable deployment model_unavailable with retry_after_secs until the next probe
Caller gone The awaiting task cancelled: the call is cancelled unless another caller waits on the id. The calling process dead (no P:caller: key): the worker cancels it on receive (caller_gone)
Cancel Waiting: never starts (cancelled); running: the attempt task is cancelled, the grant settled with nothing refunded
Deploy / restart pause_receiving() (preStop) fails readiness and stops receiving; stop(drain_secs) returns held messages and lets running calls and this pod's waiting callers finish within LLMQ_DRAIN_SECS (25). A call still running then is given back: its grants are refunded and its message returned at once, so another pod runs it, with no failure counted. A stream that already sent chunks cannot move and ends worker_shutdown (B33: 4 rolling restarts, 0 lost). See Operate
Registry invalid in the store Read leniently (unknown names written by a newer library are warnings). A stored document this pod cannot accept is refused: a running pod keeps its last good registry; a pod starting with none runs on its own document. When the pod's own registry_version is not below the stored one, the reload loop republishes its own document over the invalid one (registry_invalid_replaced, logged as an error)
Registry needs a newer library The stored document's min_lib_version is above this pod's version: the pod stays up on its last good (or own) registry, never publishes over the stored one, and logs registry_needs_newer_library
Models added by a newer registry After the reload (LLMQ_CFG_RELOAD_SECS, 30 s), the next heartbeat (every LLMQ_LEASE_SECS / 3, 20 s) starts a receive loop for each new model; a removed model's loop ends by itself and starts again if the model comes back
Watching a refused registry Every worker pod sets these gauges every 30 s: llmq_registry_invalid_reads (refused reads so far), llmq_registry_stored_refused (1 while the stored document is refused), llmq_registry_lib_too_old (1 while it needs a newer library) and llmq_registry_on_own_config (1 while the pod runs on its own document)

Every CallFailed kind is in Errors.