Skip to content

02 Stream

Stream an answer as the model writes it. Use it in chat UIs and server-sent events, where the first words should show at once.

sequenceDiagram
  participant App as App code
  participant Q as LLMQueue
  participant S as Valkey
  participant W as Worker
  participant B as Bedrock
  App->>Q: converse_stream("gpt20b", request)
  Q->>S: admit and enqueue
  S-->>W: pick
  W->>B: ConverseStream
  loop each event
    B-->>W: event
    W->>S: append to the call's chunk stream
    S-->>Q: read chunks
    Q-->>App: yield event
  end

Key points

  • converse_stream yields Bedrock ConverseStream events in order: contentBlockDelta carries the text, and metadata carries usage at the end.
  • If the stream breaks after some text was sent, CallFailed carries partial_text, and the call is not retried, so the user never sees the answer start over.
  • Waits are capped by the call's deadline, so a stream never runs past its timeout.

Code

samples/02_stream.py
"""Stream an answer as it is written (chat UIs, server-sent events to the browser).

`converse_stream` yields Bedrock ConverseStream events in order. If the stream breaks after
some text was sent, `CallFailed` carries `partial_text` and the call is not retried, so the
user never sees the answer start over.

    python scripts/samples.py 02_stream [--live]
"""

from __future__ import annotations

import asyncio

from llm_queue import CallFailed, LLMQueue, request_context


async def main() -> None:
    llmq = LLMQueue.from_env()
    await llmq.start()
    try:
        request = {
            "messages": [
                {"role": "user", "content": [{"text": "Explain a P/E ratio in two lines."}]}
            ],
            "inferenceConfig": {"maxTokens": 400},
        }
        pieces: list[str] = []
        events = 0
        try:
            with request_context(priority=True):
                async for event in llmq.converse_stream("gpt20b", request):
                    events += 1
                    delta = event.get("contentBlockDelta", {}).get("delta", {})
                    if "text" in delta:
                        pieces.append(delta["text"])
                        print(delta["text"], end="", flush=True)  # send to the client here
                    if "metadata" in event:
                        usage = event["metadata"]["usage"]
                        tokens = f"{usage['inputTokens']}/{usage['outputTokens']}"
                        print(f"\n  [{events} events, tokens {tokens}]")
        except CallFailed as exc:
            print(f"\n  stream failed: {exc.kind}; text so far: {exc.partial_text!r}")
            raise
        assert "".join(pieces).strip(), "the stream had no text"
    finally:
        await llmq.stop()


if __name__ == "__main__":
    asyncio.run(main())

Output (live, 2026-11-14)

A P/E ratio is the price of a stock divided by its earnings per share, showing how much investors are willing to pay for each dollar of earnings. It's used to gauge valuation: a low P/E may suggest a stock is cheap or undervalued, while a high P/E can signal overvaluation or high growth expectations.
  [10 events, tokens 76/142]

See also: Streaming.