Skip to content

Architecture overview

How llm-queue works, from the code under src/llm_queue. The design and its decisions are in the design plan (maintainers' notes in docs/internal, not published); a worked example with sample data is in the walkthrough (maintainers' notes in docs/internal, not published). Anything marked unverified was not confirmed in the code or by a measurement.

Purpose and boundaries

The library puts LLM calls (Amazon Bedrock, or OpenAI-compatible servers such as vLLM) through a queue on a Redis-protocol store (Valkey or Redis), and runs them under RPM / TPM budgets per deployment: one model in one account and one region. It chooses the deployment, keeps waiting calls in order, spills to fallback regions and fallback models by rule, retries by outcome, tracks deployment health, and reports status.

In scope Out of scope
Admission (budgets, concurrency slots, order), balancing, fallback, retries, health Prompt building, tool execution, agent logic (the app and LangChain do that)
Queues, results, request status, streaming to the caller, jobs (register_job) A general task queue (jobs are a small add-on: at least once, no LLM budget)
Registry of accounts, models and settings, live edits with audit Provisioning quotas at AWS; IAM; the store itself
Metrics (EMF or Prometheus), alerts as gauges and log lines, an HTTP router, a CLI Dashboards, alert routing

Every process runs the same code. The worker side runs inside the app's own pods (LLMQ_MODE=both, the default), so there is no separate worker fleet. No process keeps a count of its own: budgets, slots, waiter order and health live in the store, so 1 pod and 50 pods behave the same.

Deployment topology

flowchart TB
  subgraph pods["App pods (every pod runs the same code, LLMQ_MODE=both)"]
    direction LR
    p1["Pod 1<br/>caller + worker"]
    p2["Pod 2<br/>caller + worker"]
    pn["Pod N<br/>caller + worker"]
  end
  subgraph store["State store: Valkey / Redis, single primary + replica"]
    direction LR
    q[("Queues<br/>per model and class")]
    b[("Budgets, slots,<br/>waiters, breakers")]
    r[("Requests, results,<br/>status, registry")]
  end
  subgraph providers["Providers: one deployment = model x account x region"]
    direction LR
    d1["Bedrock<br/>account A / ap-south-1"]
    d2["Bedrock<br/>account A / ap-south-2"]
    d3["OpenAI-compatible<br/>(vLLM)"]
  end
  p1 & p2 & pn <-->|"Lua scripts:<br/>one round trip per step"| store
  p1 & p2 & pn -->|"attempts on the<br/>chosen deployment"| providers

Components

flowchart LR
  APP["App code<br/>or LangChain adapter"] --> FD["LLMQueue<br/><small>core.py</small>"]
  FD --> ENG
  subgraph ENG["Engine (engine/runtime.py)"]
    direction TB
    C["Caller side<br/>register, send, wait"]
    W["Worker side<br/>receive, admit, attempt, reply"]
    BG["Background tasks<br/>heartbeat, sweeper, prober, alerts"]
  end
  C -->|"REGISTER, CONFIRM, RESULT_TAKE"| SS[("State store<br/>LLMQ_STORE_URL")]
  W -->|"PICK_TAKE, FINISH, RESULT_PUT, STATUS"| SS
  C -->|"SEND + doorbell"| QS[("Queue store<br/>LLMQ_QUEUE_STORE_URL<br/>default: the state store")]
  W -->|"RECEIVE, EXTEND, RETURN, ACK"| QS
  W --> POOL["ClientPool + Providers<br/><small>clients/</small>"]
  POOL --> BR["Bedrock<br/>per account and region"]
  POOL --> OA["OpenAI-compatible"]
  REG["RegistryStore<br/><small>registry/</small>"] --> SS
  ENG -.reads.-> REG
Module Role
core.py Front door LLMQueue: from_env, start, pause_receiving, stop, converse / chat / call, converse_stream / chat_stream, embed, *_sync (direct path on a private loop and engine), register_job / send_job, cancel, status, request_status, health, registry, http_router
engine/runtime.py Engine: caller (register, send, wait), worker loops, held calls, attempts, retries, fallback, direct path, store_down, cancel, every background task
engine/_admission_lua.py, engine/admission.py Lua PICK_TAKE, FINISH, UNDO, RENEW, SWEEP, REREGISTER and their Python wrapper
engine/queue.py StreamQueues: a queue on a stream with visibility, generation, failures and dead letters (SEND, RECEIVE, RETURN, EXTEND, FAIL, ACK, DEPTH, REDRIVE), doorbells
engine/requests.py RequestStore: claim record, callers set, result, status (REGISTER, CONFIRM, ABANDON, RESULT_PUT, SETTLE_DONE, RESULT_TAKE, STATUS_SET, CALLERS_ALIVE)
engine/streaming.py ChunkWriter (worker) and ChunkReader (one per caller process) over P:{r:ID}:chunks
engine/status.py StatusReader (one _MODEL_STATUS script per model) and AlertEvaluator
engine/health.py Target-error counting, auto-disable, probe outcomes (RECORD, PROBED)
engine/tokens.py, engine/inference.py Token estimate and refund rule; inference defaults / limits in the provider's request shape
engine/envelope.py Message format v2 (Envelope), Reply, Meta
engine/settings.py InfraSettings.from_env: the LLMQ_* process settings
engine/store_client.py One store client factory with the failover options
engine/http.py, engine/cli.py FastAPI router (extra http); the llm-queue command
engine/jobs.py, engine/embed.py, engine/offload.py, engine/metrics.py, engine/hooks.py Jobs; embedding batching; large results in S3; EMF / Prometheus sinks; named hooks
registry/ settings.py (one table of every setting: levels, check, default), schema.py (build with every problem listed), validate.py (cross-field checks), resolve.py (ResolvedRegistry: deployments, effective values with their level), store.py + _lua.py (publish, live edit, audit, rollback, reload)
clients/ ClientHub (one SDK client per event loop, credential identity, region and options, shared by every pool in the process, ref-counted; warm connections), ClientPool (SDK retries and parameter validation off; assume-role sessions llmq-<env>-<pod>, 1 h, refreshed 5 min before expiry; pool size from the registry), Providers (Bedrock converse as plain Converse or ConverseStream per converse_api, invoke_model; OpenAI chat over SSE, embeddings), attempt timers. See Clients and connections
langchain/, http_transport.py QueuedChatBedrockConverse, QueuedBedrockEmbeddings, QueuedChatOpenAI, QueuedOpenAIEmbeddings (the OpenAI ones through an httpx transport that answers the SDK in process)
ids.py, context.py UUIDv7 ids (32 hex); request_context(priority, session_id, user_id, tenant_id)