01 Converse¶
Chat with a Bedrock model through the queue: the call most apps make most often. Use it in async request handlers (FastAPI routes), agent steps, and anything a user waits for.
sequenceDiagram
participant App as App code
participant Q as LLMQueue
participant S as Valkey
participant W as Worker
participant B as Bedrock
App->>Q: converse("gpt120b", request) in request_context(priority=True)
Q->>S: admit (budget, slot) and enqueue on the priority stream
S-->>W: pick
W->>B: Converse
B-->>W: reply + usage
W->>S: result, settle budget (real tokens)
S-->>Q: reply
Q-->>App: reply.response, reply.meta
Key points¶
LLMQueue.from_env()readsLLMQ_*;start()loads the registry and runs the workers in this process.request_context(priority=True, session_id=..., user_id=...)applies to every call inside it. Priority calls are read before batch ones.reply.responseis the Bedrock Converse body, unchanged.reply.metasays where and how the call was served: model, deployment, tokens and time spent waiting.gpt-ossmodels also return reasoning blocks;answer_textkeeps only the text blocks.
Code¶
samples/01_converse.py
"""Chat with a Bedrock model through the queue (async code: FastAPI handlers, agents, chains).
`LLMQueue.from_env()` reads `LLMQ_*`; `start()` loads the registry and runs workers in this
process. `request_context` sets priority and ids for every call inside it. The reply's
`response` is the Bedrock Converse body; `meta` says where and how the call was served.
python scripts/samples.py 01_converse # fakellm
python scripts/samples.py 01_converse --live # real Bedrock (AWS keys from .env)
"""
from __future__ import annotations
import asyncio
from typing import Any
from llm_queue import LLMQueue, request_context
def answer_text(response: dict[str, Any] | None) -> str:
"""The text blocks of a Converse reply (gpt-oss also returns reasoning blocks)."""
blocks = (response or {}).get("output", {}).get("message", {}).get("content", [])
return "".join(b["text"] for b in blocks if "text" in b)
async def main() -> None:
llmq = LLMQueue.from_env()
await llmq.start()
try:
request = {
"messages": [{"role": "user", "content": [{"text": "Name one Indian stock index."}]}],
"inferenceConfig": {"maxTokens": 300},
}
for model in ("gpt120b", "gpt20b"):
# A user-facing call: priority calls are read before batch ones.
with request_context(priority=True, session_id="s-1", user_id="u-1"):
reply = await llmq.converse(model, request)
text = answer_text(reply.response)
meta = reply.meta
print(f"{model}: {text.strip()[:120]!r}")
print(
f" served by {meta.served_model} on {meta.deployment}, "
f"tokens {meta.tokens_in}/{meta.tokens_out}, queue wait {meta.queue_wait_ms} ms"
)
assert reply.ok, reply.error
assert text, f"{model} returned no text"
finally:
await llmq.stop()
if __name__ == "__main__":
asyncio.run(main())
Output (live, 2026-11-14)¶
gpt120b: 'One well-known Indian stock index is the **NIFTY 50** (the benchmark index of the National Stock Exchange of India).'
served by gpt120b on aws-main/us-east-1, tokens 73/89, queue wait 213 ms
gpt20b: 'Nifty 50.'
served by gpt20b on aws-main/us-east-1, tokens 73/87, queue wait 48 ms
See also: Make calls, The life of a call.