02 Stream¶
Stream an answer as the model writes it. Use it in chat UIs and server-sent events, where the first words should show at once.
sequenceDiagram
participant App as App code
participant Q as LLMQueue
participant S as Valkey
participant W as Worker
participant B as Bedrock
App->>Q: converse_stream("gpt20b", request)
Q->>S: admit and enqueue
S-->>W: pick
W->>B: ConverseStream
loop each event
B-->>W: event
W->>S: append to the call's chunk stream
S-->>Q: read chunks
Q-->>App: yield event
end
Key points¶
converse_streamyields Bedrock ConverseStream events in order:contentBlockDeltacarries the text, andmetadatacarries usage at the end.- If the stream breaks after some text was sent,
CallFailedcarriespartial_text, and the call is not retried, so the user never sees the answer start over. - Waits are capped by the call's deadline, so a stream never runs past its timeout.
Code¶
samples/02_stream.py
"""Stream an answer as it is written (chat UIs, server-sent events to the browser).
`converse_stream` yields Bedrock ConverseStream events in order. If the stream breaks after
some text was sent, `CallFailed` carries `partial_text` and the call is not retried, so the
user never sees the answer start over.
python scripts/samples.py 02_stream [--live]
"""
from __future__ import annotations
import asyncio
from llm_queue import CallFailed, LLMQueue, request_context
async def main() -> None:
llmq = LLMQueue.from_env()
await llmq.start()
try:
request = {
"messages": [
{"role": "user", "content": [{"text": "Explain a P/E ratio in two lines."}]}
],
"inferenceConfig": {"maxTokens": 400},
}
pieces: list[str] = []
events = 0
try:
with request_context(priority=True):
async for event in llmq.converse_stream("gpt20b", request):
events += 1
delta = event.get("contentBlockDelta", {}).get("delta", {})
if "text" in delta:
pieces.append(delta["text"])
print(delta["text"], end="", flush=True) # send to the client here
if "metadata" in event:
usage = event["metadata"]["usage"]
tokens = f"{usage['inputTokens']}/{usage['outputTokens']}"
print(f"\n [{events} events, tokens {tokens}]")
except CallFailed as exc:
print(f"\n stream failed: {exc.kind}; text so far: {exc.partial_text!r}")
raise
assert "".join(pieces).strip(), "the stream had no text"
finally:
await llmq.stop()
if __name__ == "__main__":
asyncio.run(main())
Output (live, 2026-11-14)¶
A P/E ratio is the price of a stock divided by its earnings per share, showing how much investors are willing to pay for each dollar of earnings. It's used to gauge valuation: a low P/E may suggest a stock is cheap or undervalued, while a high P/E can signal overvaluation or high growth expectations.
[10 events, tokens 76/142]
See also: Streaming.