Skip to content

Background tasks

The loops each process runs besides the calls themselves.

flowchart TB
  subgraph caller["Caller side (LLMQ_MODE both / caller)"]
    c1["_result_reader: pub/sub wake-ups"]
    c2["_result_sweep: every 5 s"]
    c3["ChunkReader: open streams"]
  end
  subgraph worker["Worker side (LLMQ_MODE both / worker)"]
    w1["_model_loop: one per model"]
    w2["_hold: one per held call"]
    w3["_cancel_reader: cancels"]
    w4["_sweeper, _prober, _alert_loop: one process each, by lock"]
    w5["JobRunner: one per job name"]
  end
  subgraph both["Both sides"]
    b1["_heartbeat: lease / 3"]
    b2["RegistryStore.run: reload"]
  end
  STORE[("Store")]
  caller --- STORE
  worker --- STORE
  both --- STORE
Task Side Cadence Does
_model_loop (one per model) worker busy: no wait; idle: doorbell BLPOP 50 ms, 1 s after 20 empty polls; held class: local wake event receive and admit
_hold (one per held call) worker min(wait, 1 s) EXTEND + PICK_TAKE, fallback checks
_heartbeat both lease_secs / 3 (20 s), shorter for short visibility_secs; 1 s after a store error caller key, RENEW all grants, EXTEND each held message, clock sync, epoch check
_sweeper worker 10 s, one process (lock) SWEEP: drop in-progress entries and slots of owners whose lease lapsed (no refund)
_prober worker 5 s, one process per model (lock) probe auto-disabled deployments that are due (one short real call; a throttle counts as success)
_alert_loop worker 30 s, one process (lock) status report, alert gauges, a log line per flip
_cancel_reader worker pub/sub cancel running attempts named on P:cancel
_warm_clients worker once at start build every Bedrock client (first call 1,032 ms before, ~140 ms after)
_result_reader caller pub/sub wake waiters; on re-subscribe wake all
_result_sweep caller 5 s one script finds results of all waiting callers
RegistryStore.run both LLMQ_CFG_RELOAD_SECS (30) reload; republish when the stored registry was lost or older
ChunkReader caller while streams are open, XREAD block 100 ms deliver chunks
JobRunner._loop (one per job name) worker receive polls 20 ms to 500 ms run jobs, keep them visible, fail with 5 s delay