The life of a call¶
From llmq.converse(...) to the reply: who does what, on which store, in which order.
sequenceDiagram
autonumber
participant App
participant C as Caller (any pod)
participant S as State store
participant Q as Queue store
participant W as Worker (any pod)
participant P as Provider
App->>C: converse(model, params)
C->>S: REGISTER (claim record, callers set, status waiting)
C->>Q: SEND to llm-<model>[-priority] + doorbell
C->>S: CONFIRM
Note over C: waits for the wake-up, no polling
Q-->>W: doorbell wakes one idle worker
W->>Q: RECEIVE (priority queue first)
W->>S: CALLERS_ALIVE
W->>S: PICK_TAKE: deployment, RPM, TPM, slot, account budget
alt granted
W->>P: attempt on the chosen deployment
P-->>W: response + usage
W->>S: FINISH (refund to real usage, release slot)
W->>S: RESULT_PUT + PUBLISH to the caller
W->>Q: ACK
S-->>C: wake-up
C->>S: RESULT_TAKE
C-->>App: Reply(response, meta)
else refused
Note over W,S: the call is held: its class stops being read,<br/>admission is retried until granted, fallback or deadline
end
1. Resolve (memory only): alias or provider model id to alias. A model disabled at model level¶
sends the call to its first model_fallback model (if allow_fallback="all") or fails
model_disabled. timeout_secs (default 120) is capped by timeout_max_secs (900). The id is the
app's request_id or a new UUIDv7. The admission hook may Reject.
2. Route: direct (in this process) when force_direct (sync calls), routing.kill_switch, or the
id's hash falls outside queue_pct; users in routing.allow_users (HMAC hashes) always queue.
3. Register (REGISTER, one script): creates P:{r:ID}:job (state=queued, user and params
hashes, confirmed=0), adds this process to P:{r:ID}:callers, writes the status waiting. A
repeated id returns attached (wait for the first run), done, expired (result_expired),
conflict (other params or user: request_id_conflict) or takeover (an unconfirmed record older
than 5 s: its sender died).
4. Send (SEND, queue store): XADD to <queue_prefix>llm-<alias> or ...-priority (the
stream id is the store's clock: it is the call's queue position and the start of its deadline),
refused with queue_full above max_waiting (50,000), and one token pushed on the queue's
doorbell list P:bell:<queue>. With LLMQ_QUEUE_WAIT_REPLICAS the send and a WAIT go down one
pipeline; too few acknowledgements make the call run direct under the same record.
5. Confirm (CONFIRM): the record lives until the deadline plus the status TTL.
6. Wait: no read right after sending. The caller waits for the pub/sub wake-up on
P:done:<worker id>, then reads with RESULT_TAKE. A per-process sweep (_RESULTS_EXIST, every
5 s, 500 ids per script) and a re-subscribe that wakes every waiter cover missed messages.
Message states¶
stateDiagram-v2
direction LR
[*] --> Queued: SEND
Queued --> Hidden: RECEIVE (visibility_secs)
Hidden --> Held: PICK_TAKE refused
Held --> Hidden: retry admission, EXTEND
Hidden --> Running: PICK_TAKE granted
Held --> Running: granted later
Running --> Done: RESULT_PUT, ACK
Held --> Expired: max_wait_secs or deadline
Running --> Queued: worker died: visibility lapses (failure +1)
Hidden --> Queued: RETURN (newer envelope, store error)
Queued --> DeadLetter: max_receives failures
Done --> [*]
Expired --> [*]
DeadLetter --> [*]
Finish and reply¶
FINISH (refund, slot and entry removed, learning) runs first; then RESULT_PUT writes
P:{r:ID}:res once (TTL LLMQ_RESULT_TTL_SECS, 300), marks the job final and PUBLISHes the id to
every caller's P:done:<caller>; then STATUS_SET (done with deployment, served model, attempts,
tokens) and ACK (deletes the message, only with the current generation). When the queues share
the state store, a success does all of it in one script, SETTLE_DONE (result, status, ack, then
FINISH under pcall, so a failing settle never undoes the answer): one round trip instead of two.
A failure writes an error Reply the same way. RESULT_TAKE returns the result and, for a library-made id, deletes it at once
(LLMQ_RESULT_DEDUPE_SECS, default 0, keeps it that long instead); an app-supplied id keeps its TTL
so a duplicate id gets the same answer. Results above offload_bytes go to S3 (ResultOffload) and
the store keeps a marker. One llmq_call_finished log record per call (ids hashed, no text).
Direct path¶
Used for sync calls (converse_sync, embed_sync: on the started queue's loop, else on one private
loop per process with a caller-only engine; see Clients and connections),
the kill switch, queue_pct, a separate queue store that failed for LLMQ_QUEUE_FAILOVER_SECS (5),
and an unreplicated send. It registers and confirms the same record, then admits with the same
PICK_TAKE and waiter order (position = the store's time at registration; one registered waiter per
process, model and class, the rest wait on a local lock) and runs the same attempts. Only the queue
send and receive are skipped. Measured: 2,000 direct calls waiting cost 16 store scripts per process
per second (B96).