Tune¶
Timeouts, pool sizes and loop pacing are set in code with Tuning. Every value has a measured
default, so most apps pass nothing. Set only what you change:
from llm_queue import LLMQueue, StoreOptions, Tuning
llmq = LLMQueue.from_env(tuning=Tuning(store=StoreOptions(read_timeout_secs=20)))
Tuning has four parts: store (StoreOptions), bedrock (BedrockOptions), engine
(EngineTimings) and registry (RegistryOptions).
StoreOptions¶
The Valkey / Redis client of every process. The pool size is LLMQ_STORE_MAX_CONNECTIONS.
| Field | Default | What it does |
|---|---|---|
retries |
3 | Retries of a failed store command, with jittered backoff (~1.5 s in all) |
connect_timeout_secs |
5 | A store node that does not answer in a failover fails fast |
read_timeout_secs |
15 | A reply that never comes fails the command instead of hanging its call |
health_check_secs |
30 | A connection idle this long is pinged before use |
pool_wait_secs |
10 | A burst waits this long for a free connection |
BedrockOptions¶
The Bedrock and STS clients.
| Field | Default | What it does |
|---|---|---|
read_timeout_secs |
300 | Bedrock / STS read timeout (each attempt also has attempt_timeout_secs) |
connect_timeout_secs |
10 | Bedrock / STS connect timeout |
max_pool_connections |
100 | Connections per client (one client per account x region) |
role_session_secs |
3600 | Assumed-role session length (900-43200; 3600 is the role-chaining maximum) |
EngineTimings¶
Pacing of the engine's loops.
| Field | Default | What it does |
|---|---|---|
idle_poll_secs |
0.05 | Idle receive: poll this often ... |
idle_slow_secs |
1 | ... then this often ... |
idle_after_empty |
20 | ... after this many empty polls. A send's doorbell wakes a worker at once |
result_recheck_secs |
60 | A waiting caller's last-resort read of its result |
health_probe_secs |
3 | A store probe slower than this reports "down" |
stream_read_block_ms |
100 | An opened chunk stream joins the shared read within this |
admission_probe_wait_ms |
200 | A breaker's probe call is in flight: check again after this |
admission_slot_wait_ms |
50 | No free slot: retry after this ... |
admission_slot_jitter_ms |
150 | ... plus up to this, at random (spreads the retries) |
admission_wait_ms |
100 | A refusal with no better estimate retries after this |
limiter_cut_ms |
1000 | A throttle cuts the adaptive concurrency limit at most once per this ... |
limiter_cut_factor |
0.7 | ... to this fraction of itself (below 1) |
budget_key_ms |
180000 | Budget and limiter keys live this long (at least 130000: a fixed window spans 2 minutes) |
RegistryOptions¶
| Field | Default | What it does |
|---|---|---|
history_size |
20 | Registry versions kept for rollback |
edit_retries |
3 | Retries of a live edit that raced another one |
Audit retention is LLMQ_AUDIT_RETENTION_DAYS, not a tuning field.
Checks at start-up¶
from_env refuses bad tuning with ConfigError, every problem listed at once:
- wrong types, zero or negative values;
store.read_timeout_secsmust be at least 1 s aboveidle_slow_secs,LLMQ_QUEUE_WAIT_MSandstream_read_block_ms(each a blocking command), or that command times out;limiter_cut_factorbelow 1;budget_key_msat least 130000;idle_poll_secsnot aboveidle_slow_secs;role_session_secsfrom 900 to 43200 (STS limits).
Circuit breaker timing (registry)¶
The breaker is tuned per model in the registry, not in Tuning. Each setting can be set on
defaults, an account, a model or a deployment:
| Setting | Default | What it does |
|---|---|---|
breaker_threshold |
5 | Failures that open the breaker |
breaker_window_secs |
60 | Failures count toward the threshold only within this window |
breaker_cooldown_secs |
15 | An open breaker allows one probe call after this |
breaker_probe_secs |
60 | A probe call that never reported back stops blocking others after this |
How the breaker moves between states is in Failures and recovery.