Storage: keys and store usage¶
Every key the library writes, and the rules it follows on the store.
flowchart LR
subgraph reg["{reg}: registry"]
r1["doc, meta, audit, history"]
end
subgraph model["{m:MODEL}: one model"]
m1["cfg: resolved numbers"]
m2["rate:DEP: RPM / TPM clocks"]
m3["wait:p, wait:n: waiters in order"]
m4["lease, ac, cb: slots, limiter, breaker"]
m5["inprog: calls in progress"]
m6["health, sticky, retries"]
end
subgraph acct["{a:ACCT}: one account"]
a1["rate: shared budgets"]
end
subgraph req["{r:ID}: one request"]
q1["job, callers, status"]
q2["res, cancel, chunks"]
end
subgraph queue["{q:NAME}: one queue"]
s1["s: stream of messages"]
s2["vis, gen, fails, dlq"]
end
PICK(["PICK_TAKE"]) --> model & acct & req
SEND(["SEND"]) --> queue
REG(["REGISTER_SEND / RESULT_PUT"]) --> req & queue
P is LLMQ_KEY_PREFIX; each box is one hash tag. One admission script touches several
tags at once, which is why cluster mode is not supported (see Store usage).
Key layout¶
P is LLMQ_KEY_PREFIX (default llmq). Queue names are LLMQ_QUEUE_PREFIX + llm-<alias> (or the
model's queue.name) and ...-priority; jobs use jobs-<name>. Hash tags group a model
({m:A}), a request ({r:ID}), a queue ({q:NAME}), an account ({a:ACCT}) and the registry
({reg}), but scripts cross tags (see Storage).
| Keys | Type | Holds | Lifetime |
|---|---|---|---|
P:{reg}:doc, :meta, :audit, :history |
string, hash, stream, hash | Registry JSON; version, config_hash, changes, published; one audit entry per publish / edit (actor, op, version, diff, ts); the last 20 published versions |
audit trimmed to LLMQ_AUDIT_RETENTION_DAYS (90) |
P:{m:A}:cfg |
hash | numbers (resolved admission settings per deployment), changes |
until registry gc |
P:{m:A}:rate:DEP |
hash | rpm_tat, tpm_tat, class clocks p_/n_rpm_tat, p_/n_tpm_tat (or *_win / *_n when fixed); seen_win, seen_calls, seen_tokens, seen_n |
180 s, refreshed on write |
P:{a:ACCT}:rate |
hash | Account budget clocks / counters | 180 s |
P:{m:A}:wait:p, :wait:n, :waitinfo, :waitexp |
zset, zset, hash, zset | Waiters by queue position; their cost, usable deployments, expiry; waiters by expiry (so PICK_TAKE touches only expired ones) |
entries expire after 2 s unless refreshed |
P:{m:A}:ac:DEP, :lease:DEP:p, :lease:DEP:n, :cb:DEP |
hash, zsets, hash | Learned concurrency (limit, cut_at); slots in use (owner to lease expiry); breaker (state, failures, until, probe, probe_at) |
no TTL (leases expire by score) |
P:{m:A}:inprog, :inprog:ID |
zset, hash | Calls in progress (owner to lease expiry); the grant (owner, worker, dep, cls, minute, cost, account, started) | entry 2 x lease, renewed |
P:{m:A}:health, :terr:DEP, :thr:DEP:MIN |
hash, zset, counter | fields DEP\|until, DEP\|wait, DEP\|since, DEP\|lat (latency EWMA, ms), DEP\|n; target errors by request id; throttles this minute |
window; 180 s |
P:{m:A}:rot, :sticky:HASH, :retries:MIN |
hash, string, hash | Rotation; a session's deployment; retries and first attempts this minute (first attempts counted by PICK_TAKE at grant) |
none; sticky.idle_ttl_secs (1800); 180 s |
P:{r:ID}:job, :callers, :status, :res, :cancel, :chunks, :debug |
hash, set, hash, string, string, stream, string | Claim; calling processes; status (state, model, priority, <state>_at, ...); result; cancel flag; stream; debug_capture text |
job and callers: deadline + status TTL; status: LLMQ_STATUS_TTL_SECS (1800); res: LLMQ_RESULT_TTL_SECS (300), deleted on first read for a library id; cancel: status TTL; chunks: result TTL + timeout; debug: 1 h |
P:{q:NAME}:s, :vis, :gen, :fails, :st, :dlq, :meta |
stream, zset, hashes, stream, hash | Messages; hidden until; generation; failures; held / returned; dead letters (body, src, fails); the receive cursor |
stream MINID 1 day (retention_secs); DLQ 1 day |
P:bell:NAME |
list | Doorbell tokens (at most 64) | 60 s |
P:caller:WORKER, P:done:WORKER, P:cancel, P:chunkwake:WORKER |
string, channel, channel, stream | Process liveness; result wake-ups; cancels; wakes the process's shared chunk reader when a new stream opens | LLMQ_CALLER_TTL_SECS (30); chunkwake trimmed |
P:epoch, P:reset_minute, P:lock:{sweep,alerts,probe}, P:alert:*, P:health:WORKER |
strings | Store-reset detector; locks for one-process tasks; alert state; readiness probe | locks 5-25 s; alerts 1 day; probe 5 s |
Store usage¶
- Single primary only (with replicas). Cluster mode is not supported (
start()refuses it, see Store requirements): the scripts build key names inside Lua from the prefix rather than taking them inKEYS, and one script touches several tags (PICK_TAKEwrites{m:},{r:}and{a:}keys;SENDpushes the doorbell, which is not inKEYSand has no tag). Two stores are possible: the queue store holds only queues and doorbells. - Time is the store's (
TIMEinside every script); a process keeps an offset (StoreClock, resynced every heartbeat), so pod clock skew does not matter. - Client (
store_client.py): bytes replies,BlockingConnectionPoolwithLLMQ_STORE_MAX_CONNECTIONS(200) and a 10 s wait for a free connection, connect timeout 5 s, keepalive, idle health check 30 s, and 3 retries with jitter (about 1.5 s) on dropped connections, timeouts andREADONLY.READONLYcloses the connection, so the retry resolves the DNS name again and reaches the new primary after a failover (before, every write failed until a restart).valkey:///valkeys://are accepted. Store passwords can come fromfrom_env(secrets=), read on every new connection. - Readiness (
health_check): ready means registry loaded and not draining. A background task probes each store every 5 s with a short write bounded byEngineTimings.health_probe_secs(3 s) and caches the answer, so a readiness probe answers at once. Each store is reportedok,read_onlyordown(a refused write, such as out of memory, isdown); the stores' state is reported but never makes the process unready, and a store error never turns into a 500. Liveness is false when a background loop died or the heartbeat stopped for a lease. NOSCRIPT: a pipelinedSEND+WAITthat meets a node without the script cached is run once more (pipelines load scripts only when they start); plain script calls rely on redis-py's reload.- Queue store failover:
LLMQ_QUEUE_FAILOVER_SECS(5) of send errors (each try bounded by it) before a call runs direct; the next calls skip the queue store while the outage is remembered;LLMQ_QUEUE_WAIT_REPLICAS/LLMQ_QUEUE_WAIT_MS(0 / 100) make a send wait for replicas. - Store reset: a missing or changed
P:epoch(restart without data, failover to an empty node) is detected by every pod's heartbeat (each remembers the epoch it saw): each pod puts its own running grants back (REREGISTER), and grantsRENEWreports as lost are registered again. A fixed window holds until the next minute, and the reload loop republishes the registry (live edits and live blocks are lost with the store's data, and logged). - TLS: unless
LLMQ_ENVIRONMENTisdev/local/test/ci(unset counts as strict), store URLs must berediss://with certificate and host-name checks on (a localhost URL is allowed), andLLMQ_ID_HASH_SALTis required. The URL is read with redis-py's own parser, a repeated query parameter is refused, andssl_cert_reqs=required/ssl_check_hostname=trueare written into the URL explicitly. See Environment variables.
Store requirements¶
| Requirement | Why | What the library does |
|---|---|---|
maxmemory-policy noeviction (recommended) |
Any evicting policy lets a full store drop queue messages, budgets and results: lost calls and overspent budgets | Reads the policy at start() and every minute after (CONFIG GET, else INFO memory where CONFIG is disabled, as on ElastiCache); logs store_eviction_policy with a warning when it is not noeviction and, in strict mode, fails readiness with not_ready_reason; reports it in health(): eviction_policy for the state store (unknown when unreadable), queue_eviction_policy only when the queue store is a separate one |
| One primary (with replicas), not cluster mode | One admission script touches keys of several hash tags | start() refuses a cluster-mode store with a ConfigError (INFO cluster) instead of failing at the first call |
Lua scripts, pub/sub, blocking list reads (BLPOP) |
Admission, result wake-ups and the queue doorbell | ElastiCache for Valkey / Redis OSS allows all three |
| A store that is out of memory | noeviction refuses writes when full |
Calls fail queue_unavailable ("out of memory"), admin operations StoreUnavailable; nothing is lost silently |
min-replicas-to-write 1 (recommended with replicas) |
Without it a primary that lost its replicas keeps accepting writes, and a failover then drops what only it held | A write the store refuses for it (NOREPLICAS), like OOM, MISCONF (a failed save), BUSY (a long script) and READONLY, is treated as the store being down: store_down decides (pause, static cap, fail fast); a result already written stays readable |
Set min-replicas-to-write with min-replicas-max-lag (for example 1 and 10 s) on a replicated
self-managed Valkey or Redis; on a managed service, check whether its parameter group exposes
them (unverified for ElastiCache). LLMQ_QUEUE_WAIT_REPLICAS adds a WAIT after each
send for the same guarantee per message.
Tested versions: Valkey 8.1 and Redis 7.4 (the docker/compose.yaml stack every integration
test, scenario and gate runs on); Valkey 8 and Redis 7.2+ are supported. The client is redis-py
6 or later (redis>=6,<9): from 6 on, TLS connections check the host name by default, and the
URL parser the settings check uses is the one the client uses.