End-to-end samples¶
Nine runnable programs, one per way an app uses llm-queue. Each is plain app code: it builds
the queue with LLMQueue.from_env() and reads nothing else, so you can copy it into a service
as is. Every page shows the full code and the real output of a live run on Amazon Bedrock.
Last live run: 2026-11-14, code v0.1.0b2
All nine samples passed on real Bedrock in us-east-1, with an ElastiCache Valkey 8.2 (TLS)
as the store, in strict mode (LLMQ_ENVIRONMENT unset): python scripts/samples.py --live
--store env. The run used its own key prefix and left no keys behind.
What a sample run does¶
flowchart LR
subgraph runner["scripts/samples.py"]
env["sets LLMQ_* env<br/>(store, prefix, registry)"]
end
subgraph proc["Sample process (your app code)"]
q["LLMQueue.from_env()<br/>start()"]
call["converse / stream / embed<br/>jobs / status / router"]
w["worker loop<br/>(same process)"]
end
store[("Valkey / Redis<br/>queues, budgets, health")]
llm["Amazon Bedrock<br/>(or fakellm)"]
env --> q --> call
call -- "admit + enqueue" --> store
store -- "pick" --> w
w -- "Converse / InvokeModel" --> llm
w -- "result" --> store
store -- "reply" --> call
The runner only sets the environment and starts each sample as its own process. Inside, the
sample starts the queue; the same process acts as caller and worker (LLMQ_MODE=both), just as
an app pod does. At the end the runner deletes every key under the run's prefix.
The samples¶
| Sample | Shows | Use it in |
|---|---|---|
| 01 Converse | converse with request_context; reply text and meta |
Async request handlers, agent steps, anything user-facing |
| 02 Stream | converse_stream events as they arrive; partial_text on a cut |
Chat UIs and server-sent events |
| 03 Embed | embed on Titan (1 text per request) and Cohere (96, input_type) |
Indexing documents and search queries |
| 04 Sync | converse_sync / embed_sync on the direct path |
Scripts, Celery tasks, sync libraries |
| 05 Errors and fallback | Model fallback; CallFailed.kind to branch on |
Every caller |
| 06 Jobs | register_job / send_job, LLM calls inside a job |
Ingestion, nightly re-embedding, batch work |
| 07 Status and registry | status, request_status, live registry.set, audit log, health |
Admin pages, runbooks, readiness checks |
| 08 LangChain | QueuedChatBedrockConverse (with tools), QueuedBedrockEmbeddings |
LangChain / LangGraph chains, agents, RAG |
| 09 HTTP router | http_router on FastAPI: probes, admin routes, /drain |
Admin API, Kubernetes probes, the preStop hook |
The registry they share¶
samples/registry.json defines four models on one AWS account: chat gpt120b (falls back to
gpt20b) and gpt20b, and embeddings titan and cohere. The quotas are small sample
numbers; use your Service Quotas numbers in a real registry. The runner sets every deployment
and the residency list to the run's region.
{
"registry_version": 1,
"defaults": {"timeout_secs": 120, "attempt_timeout_secs": 60, "throttle_retry_attempts": 6},
"residency": {"allowed_regions": ["us-east-1"]},
"accounts": {"aws-main": {"kind": "aws"}},
"models": {
"gpt120b": {
"type": "chat", "provider": "bedrock", "model_id": "openai.gpt-oss-120b-1:0",
"rpm": 60, "tpm": 100000, "balance_priority": "ordered",
"inference": {"max_tokens": 512},
"model_fallback": {"on": ["exhausted", "broken"], "after_secs": 5,
"models": [{"model": "gpt20b", "require_headroom": true}]},
"deployments": {"aws-main": {"confirmed": ["us-east-1"]}}
},
"gpt20b": {
"type": "chat", "provider": "bedrock", "model_id": "openai.gpt-oss-20b-1:0",
"rpm": 60, "tpm": 100000,
"inference": {"max_tokens": 512},
"deployments": {"aws-main": {"confirmed": ["us-east-1"]}}
},
"titan": {
"type": "embedding", "provider": "bedrock", "model_id": "amazon.titan-embed-text-v2:0",
"rpm": 120, "embed_batch_size": 1,
"deployments": {"aws-main": {"confirmed": ["us-east-1"]}}
},
"cohere": {
"type": "embedding", "provider": "bedrock", "model_id": "cohere.embed-english-v3",
"rpm": 120, "embed_batch_size": 96,
"deployments": {"aws-main": {"confirmed": ["us-east-1"]}}
}
}
}
Run them¶
Start the local stack once (Valkey, plus fakellm as a fake Bedrock), then run from the library root:
python scripts/stack.py up
python scripts/samples.py # all, on fakellm (no real LLM, no cost)
python scripts/samples.py 03_embed 08_langchain # some
python scripts/samples.py --live --env-file ../../.env # real Bedrock, AWS keys from that file
python scripts/samples.py --live --store env --env-file ../../.env # and that file's store
- What
--livereads:AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY,AWS_SESSION_TOKENandAWS_MODEL_REGIONfrom the env file. It never prints them. - Cost: a full live run makes about 20 small calls, well under a cent.
--store env: uses the env file'sLLMQ_STORE_URL. Only the run's own prefix is touched and deleted, so a shared store (ElastiCache) is safe.- In CI:
python scripts/check.py samplesruns the fakellm pass with lint and type checks.
Use them in an app¶
An app sets the same environment the runner sets, from its own secrets:
| Variable | Runner value | In an app |
|---|---|---|
LLMQ_ENVIRONMENT |
test with the local Valkey; unset with --store env |
unset or your environment's name (strict) |
LLMQ_ID_HASH_SALT |
a random one per run with --store env |
from your secrets (required when strict) |
LLMQ_STORE_URL |
redis://:localdev@127.0.0.1:6379/0 |
rediss://... of the ElastiCache Valkey |
LLMQ_KEY_PREFIX, LLMQ_QUEUE_PREFIX |
per run | per environment, e.g. llmq:prod |
LLMQ_REGISTRY |
samples/registry.json, region set |
the app's registry document |
LLMQ_METRICS_SINK |
none |
emf (CloudWatch) or prometheus |
| AWS credentials | from .env (or fake for fakellm) |
the pod's role; role_arn accounts assume roles |
Start the queue once per process in the app's startup hook, and stop it on shutdown:
@asynccontextmanager
async def lifespan(app):
await llmq.start() # a failure here should fail the pod: queued models need it
yield
llmq.pause_receiving() # not ready; no new queued work (also from a preStop /drain)
await llmq.stop() # LLMQ_DRAIN_SECS (25 s) for running jobs and calls together
Unless LLMQ_ENVIRONMENT is dev, local, test or ci (unset counts as strict),
LLMQ_STORE_URL must be rediss:// and LLMQ_ID_HASH_SALT must be set.
Measured on real Bedrock¶
| What | Result |
|---|---|
| Models | gpt-oss-120b, gpt-oss-20b, Titan v2 and Cohere v3 all answered through the queue (us-east-1 and ap-south-1) |
| Embeddings, similar / unrelated pair | Titan v2: 0.73 / 0.07; Cohere v3: 0.83 / 0.37 (1024 dimensions each) |
| Streaming overhead | a warm gpt-oss-20b stream took 0.83 s through the queue against 0.7 s called directly |
| Queue wait (2026-11-14 run) | 48 to 213 ms for a priority call on an idle queue over TLS ElastiCache |
| Fallback | with gpt120b off, the call went to gpt20b (meta.fallback == "model") |
| Tool calls | LangChain returned stock_price({"symbol": "RELIANCE"}) |