Skip to content

Configuration

Where the process settings (LLMQ_*) are listed, the Tuning fields set in code, and where the registry settings are listed. How to use them is in Configure and Tune.

Process settings (LLMQ_*)

Every LLMQ_* variable, its default, and what LLMQ_ENVIRONMENT turns on (unset is strict) are in Environment variables.

Jobs have their own process settings, LLMQ_JOB_* (read by LLMQueue.from_env, or passed as LLMQueue(..., job_settings=JobSettings(...))); a bad value is a ConfigError at start:

Variable Default Meaning
LLMQ_JOB_DEDUPE_SECS 86400 (24 h) send_job(job_id=X) again within this queues nothing and returns X; 0 turns it off
LLMQ_JOB_RETRY_BASE_SECS 5 First retry delay; it doubles after each failure ...
LLMQ_JOB_RETRY_MAX_SECS 300 ... up to this (at least the base)
LLMQ_JOB_RETRY_FOR_SECS 3600 Retries go on until the delays add up to this; then the job is dead
LLMQ_JOB_DEAD_RETENTION_SECS 604800 (7 days) How long dead jobs are kept (60 to 1209600)
LLMQ_JOB_CONCURRENCY 4 Running jobs per job name per process (register_job(..., concurrency=N) wins)
LLMQ_JOB_PROCESS_CONCURRENCY 32 Running jobs per process, every name together
LLMQ_JOB_MAX_PAYLOAD_BYTES 1048576 (1 MB) Largest job message; a larger send fails request_too_large (at least 1024)
LLMQ_JOB_MAX_RETURNS 5 Returns at shutdown before a job is dead-lettered; 0 never

See Run background jobs.

Tuning (in code)

Passed as LLMQueue.from_env(tuning=Tuning(...)). Defaults:

Part Field (default)
StoreOptions (tuning.store) retries (3), connect_timeout_secs (5), read_timeout_secs (15), health_check_secs (30), pool_wait_secs (10)
BedrockOptions (tuning.bedrock) read_timeout_secs (300), connect_timeout_secs (10), max_pool_connections (100: used only until the registry sizes the pool, see below), role_session_secs (3600), transport ("aiohttp": orjson body, aiohttp session, botocore only for credentials, SigV4 signing and stream frames; "botocore": the aiobotocore client)
EngineTimings (tuning.engine) idle_poll_secs (0.05), idle_slow_secs (1), idle_after_empty (20), result_recheck_secs (60), health_probe_secs (3), stream_read_block_ms (100), admission_probe_wait_ms (200), admission_slot_wait_ms (50), admission_slot_jitter_ms (150), admission_wait_ms (100), limiter_cut_ms (1000), limiter_cut_factor (0.7), budget_key_ms (180000), throttle_base_ms (1000), throttle_max_ms (30000), slow_stale_ms (300000), job_ttl_ms (3600000)
RegistryOptions (tuning.registry) history_size (20), edit_retries (3)
OpenAIOptions (tuning.openai) read_timeout_secs (300), connect_timeout_secs (10), max_connections (100), max_keepalive_connections (20), keepalive_secs (15), http ("aiohttp": the SDK's aiohttp client with the same limits and timeouts, except max_keepalive_connections, which aiohttp has no equivalent for; falls back to httpx with a warning when the SDK has none)
WarmOptions (tuning.warm) connections (2; 0 = off), refresh_secs (10; 0 = warm once, no repeats), keepalive_secs (15)

What each field does and the start-up checks are in Tune.

Bedrock connection pools are sized from the registry: each (account, region) client gets the sum of the concurrency slots routed to it, at least 10 connections (no upper cap). WarmOptions (2 warm connections per client, refreshed every 10 s, kept idle 15 s) controls the warm connections; see Clients and connections. Both are importable from llm_queue and passed like the other parts:

from llm_queue import LLMQueue, OpenAIOptions, Tuning, WarmOptions

queue = LLMQueue.from_env(
    tuning=Tuning(openai=OpenAIOptions(read_timeout_secs=120), warm=WarmOptions(connections=4))
)

Registry settings

The registry document's settings can each be set at some of the levels deployment > model > account > defaults (the most specific one wins; objects marked merged combine key by key). Bounds: every number is finite and at most 2**53 - 1 (9007199254740991); every timeout and wait is at most one day (86400 s); tighter bounds are listed below. A value outside its bound is refused with its path when the registry is loaded or edited.

Setting Levels Default Range and meaning
enabled account, model, deployment true Off: no new calls (live edit; llm-queue drain sets it)
rpm, tpm defaults, model, deployment none Requests / tokens per minute of a deployment; whole numbers of at least 1
rate_limits all four chat both, embedding rpm Which budgets apply: both, rpm, tpm or off
rate_window all four smooth smooth (sliding) or fixed (calendar minute)
rate_burst all four 0.1 Share of a minute's budget usable at once, 0 to 1
tpm_output_weight all four 1 Output tokens count this many times against TPM, 1 to 1000
tpm_count_cached all four false Cache reads count against TPM
default_output_tokens all four 1000 Output reserved for a call that sets no max_tokens, 1 to 10000000
shared_rpm, shared_tpm account none One budget shared by every deployment of the account
max_concurrency all four none (200 in status) Calls in flight per deployment, 1 to 100000
priority_reserve defaults, account, model chat 0.2, embedding 0 Share kept for priority calls, 0 to 1
normal_min_share defaults, model 0.1 Share normal calls always get, 0 to 1; with priority_reserve below 1
timeout_secs defaults, account, model 120 A call's deadline; at most timeout_max_secs
timeout_max_secs defaults, model 900 The largest timeout_secs a caller may ask for
max_wait_secs defaults, model none Longest wait for budget before rate_limited
attempt_timeout_secs all four 60 One provider attempt; below timeout_secs
throttle_retry_attempts all four 6 Retries of a throttled attempt, 1 to 20
retry_base_secs, retry_multiplier, retry_spread, retry_max_secs all four 1, 4, 2.5, 30 Backoff: base x multiplier^n, spread, capped; multiplier and spread 1 to 100
breaker_threshold all four 5 Failures that open a deployment's breaker, 1 to 10000
breaker_cooldown_secs, breaker_window_secs, breaker_probe_secs all four 15, 60, 60 Breaker timing; the probe at least attempt_timeout_secs
balance defaults, model ordered ordered, round_robin, weighted, headroom or custom:<name>
balance_opts defaults, model {"pick": "percent"} merged; pick: percent, max or spread
balance_priority model as balance The strategy for priority calls
weight deployment 1 Share under weighted; at least one usable deployment must be above 0
tags deployment none Text or number labels a custom balancer reads
max_attempts defaults, model 4 Attempts per call across deployments, 1 to 100
retry_budget_pct, retry_budget_min defaults, model 20, 10 Retries allowed as a share of calls, with a floor
stream_idle_secs defaults, model 30 A stream with no event this long is cut
stream_max_bytes model 8388608 Largest streamed answer, 1024 to 1073741824
stream_flush_ms model, deployment 30 Chunk batching, 0 to 1000
max_message_bytes defaults, model 4194304 Larger messages fail request_too_large before they are queued, 1024 to 268435456
converse_api defaults, model converse Bedrock non-stream chat: converse or stream
supports_tools model true False: a call with tools served by this model as a fallback loses its tool fields
max_waiting defaults, model 50000 Calls a queue holds before queue_full; 0 means no limit, not "do not queue"
queue_pct defaults, model 100 Share of users (by id hash) whose calls are queued; the rest run direct (a canary), 0 to 100
worker_slots defaults, model 20 Calls a worker runs at once per model, 1 to 1000 (deploy only)
queue defaults, model see below merged: name (model only; 1-71 letters, digits, _, ., -, not starting jobs-), visibility_secs (60, 1 to 43200), max_receives (10, 1 to 1000), retention_secs (86400, one day, 60 to 1209600), dlq_retention_secs (86400, 60 to 1209600)
sticky defaults, model none {"key": "session" or "custom:<name>", "idle_ttl_secs"}
sticky_move_after_secs defaults, model 1 A sticky session keeps its deployment while it would wait at most this long
require_headroom_pct defaults, model 20 Free share a fallback target listed with require_headroom needs, 0 to 100
fallback defaults, model none Fallback regions: {"on": [exhausted, broken, error], "after_secs" (2), "regions": account or any}
model_fallback model none Fallback models (protected): {"on", "after_secs" (5), "models": [alias or {model, require_headroom}]}. A target may use another provider: converted
store_down defaults, model pause pause, static_cap or fail_fast
embed_batch_size model 1 Texts per embedding request, 1 to 4096 (embedding models only)
embed_max_chars model the family's (Titan 50000, Cohere 2048) Longest text an embed sends, 0 to 10000000 (0: off; embedding models only)
inference all four none merged defaults of the request: max_tokens, temperature, top_p, stop, top_k, seed, penalties, extra; embedding dimensions, normalize, input_type, truncate, embedding_types
inference_limits all four none merged: max_tokens cap (also max_completion_tokens; a call without one gets the cap), locked names, on_exceed (clamp or reject)
params deployment none Extra request fields of one deployment (guardrails); protected
health all four see below merged: disable_errors (3), disable_window_secs (300), probe_after_secs (900), probe_max_secs (7200), slow_ratio (3), slow_min_secs (5), slow_min_samples (20)
alerts defaults, model {"quota_used_pct": 90, "dlq_messages": 1, "queue_age_secs": 600} merged; queue_age_secs (1 or more) turns queue_stalled on when the oldest queued call is older; the four names removed in 0.1.0b2 are ignored with a warning until 0.2
admission, fallback_policy, token_estimator, probe defaults, model none Names of registered hooks (1-64 letters, digits, _, ., -)
debug_capture model false Store prompt and answer for an hour (dev only; protected)
fairness defaults, model {"mode": "off", "default_weight": 1, "active_secs": 60} merged; per-tenant fairness, see Tenant fairness. mode: off, share, quota or fair_queue; weights (share: tenant id -> weight above 0, at most 1000000) and default_weight; rpm, tpm (quota: each tenant's cap, whole numbers of at least 1) and quotas (tenant id -> {rpm, tpm}); active_secs (1 to 86400). Tenant ids 1-256 characters. A quota with no cap is refused; a cap above the model's budget, a tpm cap below one call's reservation and settings the mode never reads warn

Names (aliases, accounts, regions) are 1-64 lowercase letters, digits, _, . or -, starting with a letter or digit; a trailing newline or padding is refused. A stored document is read leniently: an unknown setting or sub-field from a newer library is ignored with a warning, while the process's own LLMQ_REGISTRY stays strict.

For a running system, llm-queue settings get --model M [--deployment D] prints the effective value of every setting and the level it came from.

Settings added or changed in 0.1.0b2:

Setting Levels Default Meaning
converse_api defaults, model converse Non-stream Bedrock chat: converse (plain Converse, IAM bedrock:InvokeModel) or stream (ConverseStream, bedrock:InvokeModelWithResponseStream, with the first-response timer). See Bedrock APIs and IAM
max_message_bytes defaults, model 4194304 (4 MB) A call whose message (inline images and documents included) is larger fails request_too_large before it is queued; 1024 to 268435456
inference.extra as inference Provider-specific fields: plain lowercase names only; SDK arguments and request fields (extra_body, store, metadata, user, model, messages, stream, tools, ...) are refused. Protected in safe live-edit mode
queue.retention_secs, queue.dlq_retention_secs one day each Now at most two weeks (1209600): messages and dead letters hold the prompt. retention_secs defaults to one day (was four days) and is applied on every send
embed_max_chars model (embedding) the family's: Titan 50000, Cohere 2048, OpenAI none Longest text an embed sends; a longer one fails invalid_request with its index before any call. 0 turns the check off
family model field (embedding) openai for the openai provider, else from model_id titan, cohere, openai or a name added with register_embedding_family: the request and reply shape. Required when the id names no family (an application inference profile ARN); the registry load fails without it
inference.embedding_types as inference Only ["float"]: the other types were paid for, then unreadable
alerts quota_used_pct (1-100, default 90; quota_near_limit now checks TPM too), dlq_messages (1 or more, default 1) and the new queue_age_secs (default 600, for queue_stalled). quota_used_mins, spill_pct, spill_window_secs and waiting_age_secs did nothing and were removed: a document naming them still loads, with a deprecation warning, until 0.2
live_edit (top level) document unset The document's live-edit mode (deploy-only)
min_lib_version (top level) document unset Now a version such as 0.1.0b2; checked against the running library

Validation added in 0.1.0b2 (each refused with its path): a TPM below one call's reservation (max output x tpm_output_weight); a fallback cycle of any length (unless every step requires headroom); a fallback target whose residency is not within the source model's; residency that leaves a model with no deployment (an error, no longer a warning); breaker_probe_secs below attempt_timeout_secs; a fallback after_secs that could never fire inside timeout_secs; NaN, infinities and numbers above their bound; names with a trailing newline or padding; malformed role ARNs; queue names that are not plain or start with jobs-.

Warnings (the document loads; each is logged with its path): a fallback to a disabled model; a blocked_regions entry no deployment uses or a blocked_accounts entry that is not an account; a model with no usable confirmed deployment; queue retention below the timeout; a retry base above its cap; probe_after_secs above probe_max_secs; duplicate fallback causes or models; a fallback rule with no fallback regions; settings the model's type or provider never reads; a model_fallback to a model of another provider (allowed: see below).

Fallback to a model of another provider

A model_fallback target may be a model of the other provider: a Bedrock (Converse) model may fall back to an OpenAI-compatible model and the other way round. The caller still writes its request in its own model's shape and reads the reply in that shape; the worker converts both:

Part Converse OpenAI chat
System prompt system text blocks a system (or developer) message
Text and images text, image (inline bytes: png, jpeg, gif, webp) string content, text and image_url parts (base64 data URLs)
Tool use toolUse blocks in an assistant turn assistant tool_calls (arguments as JSON text)
Tool results toolResult blocks in a user turn (status: error becomes an error: prefix) tool messages
Tools toolConfig.tools[].toolSpec (inputSchema.json) tools[].function (parameters)
Tool choice toolChoice auto / any / tool tool_choice auto / required / a named function (none has no Converse counterpart: left out)
Output and sampling inferenceConfig.maxTokens, temperature, topP, stopSequences max_tokens (or max_completion_tokens), temperature, top_p, stop
Reply output.message (text, toolUse), stopReason, usage (input, output, total, cache read) choices[0].message (content, tool_calls), finish_reason, usage (prompt, completion, total, cached)

Stop reasons map end_turn / stop_sequence to stop, max_tokens to length, tool_use to tool_calls and the guardrail / content filter reasons to content_filter (and back). Fields with no counterpart are dropped: Converse additionalModelRequestFields, guardrails, cache points and reasoning blocks of past turns; OpenAI response_format, logprobs, n, penalties and seed. The target's own inference defaults and limits apply after the conversion.

A request the other shape cannot carry (a document or video block, an image by URL or S3 location instead of inline bytes, an image inside a tool result, a non-text OpenAI part such as audio, tool arguments that are not JSON) skips that target (logged fallback_shape_skipped); the call then waits, tries the next target or fails as it would without it. Streams do not fall back across shapes: their events are not converted, so a stream skips such a target (a disabled model whose only targets are of the other shape fails a stream with model_unavailable). The validator warns where such a target is configured.

LLMQ_CALLER_TTL_SECS must be at least LLMQ_LEASE_SECS / 3 + max(5, LLMQ_LEASE_SECS / 6). The registry audit is capped at about 10,000 entries besides its age limit.