Many services, one model¶
Several services, each with many pods, can call the same models through the same queues and budgets. Budgets, slots, waiter order and health live in the store, so the limits hold across every process that shares it.
flowchart LR
subgraph A["Service A (pods 1..N)"]
a1["caller + worker"]
end
subgraph B["Service B (pods 1..M)"]
b1["caller + worker"]
end
A & B <--> S[("One store<br/>same LLMQ_KEY_PREFIX / LLMQ_QUEUE_PREFIX")]
S --> M["Model gpt120b<br/>one RPM / TPM budget per deployment"]
What the services must share¶
| Must be the same | Why |
|---|---|
The store (LLMQ_STORE_URL, and LLMQ_QUEUE_STORE_URL if used) |
Queues, budgets and results live there |
LLMQ_KEY_PREFIX and LLMQ_QUEUE_PREFIX |
They name the keys and queues; a different prefix is a separate system with its own budgets |
| The registry document | Every service ships the same document (LLMQ_REGISTRY or LLMQ_REGISTRY_FILE) |
A service in caller mode only makes calls; the workers of any service in both or worker
mode run them.
The registry version rule¶
Each process publishes its registry at start() by registry_version:
| This process's version vs the store's | What happens |
|---|---|
| Higher | It is published and replaces the stored one; live routing blocks and the kill switch of the stored one are carried over |
| Equal, same content | The stored one is kept, with its live edits |
| Equal, other content | The stored one is kept; the process logs registry_version_not_bumped |
| Lower | The stored one is used; the process logs registry_older_ignored |
| Stored one invalid for this pod, its version not above this one | This pod's document replaces it (registry_invalid_replaced) |
Stored one needs a newer library (min_lib_version) |
Never replaced; this pod keeps its own document and logs registry_needs_newer_library |
So during a rolling deploy of one service, its new pods publish the new version and the old pods
of every service pick it up on their next reload (LLMQ_CFG_RELOAD_SECS, 30 s). Raise
registry_version with every deployed change, and ship the new document to every service.
What a service on an older document gets from a newer stored one:
- Changed settings and deployments of models it knows: used at the next reload.
- New models: after the next reload, running pods start a receive loop for each new model on
their next heartbeat (every
LLMQ_LEASE_SECS/ 3), so their calls are served; the service's own code can call them by alias once it knows the name. - Removed models: the model's receive loop ends by itself; it starts again if a later registry brings the model back.
- Settings it does not know (written by a newer library): read as warnings and ignored by this pod; the newer pods apply them.
- A
min_lib_versionabove its library: the pod stays up on its own registry, never publishes over the stored one, and warns. Upgrade the library.
Emergency routing blocks (llm-queue block) and the kill switch are carried into every publish:
a deploy never undoes them; only an explicit unblock does.
flowchart TD
S([start / reload]) --> R{Stored registry}
R -- "missing" --> P[Publish own document]
R -- "needs newer library" --> K[Keep own / last good<br/>never publish, warn]
R -- "invalid for this pod" --> V{Own version >= stored?}
V -- yes --> P
V -- no --> K
R -- "valid" --> C{Own version vs stored}
C -- higher --> P2[Publish, carry live blocks]
C -- "equal / lower" --> U[Use the stored one]
Fairness between clients¶
Calls are served in arrival order within a class (priority, then normal). There is no fair share per client or per service: one client that floods a model delays every other client of that model.
To keep latency-sensitive callers moving:
- mark user-facing calls
priority=True(or withrequest_context(priority=True)); priority calls are read first, andpriority_reservekeeps part of each budget for them; - give callers that should not wait long a
max_wait_secs, so they failrate_limited(withretry_after_secs) instead of waiting behind the flood.
Library versions¶
Services may run different llm-queue versions for a while. Queue messages carry a format
version; a worker that gets a message newer than it understands returns it to the queue for
15 s with no failure counted, never drops it, so a newer worker can take it. Each one counts
llmq_newer_messages_total, with one warning a minute. Set the registry's
min_lib_version when a registry change needs a newer library: older pods then stay up but
never publish over that registry. See the stability policy.
Each service also needs its own process settings right: the same LLMQ_ENVIRONMENT class
(strict or dev) everywhere, since a strict process refuses plain-TCP stores and a missing salt
(Environment variables).