Skip to content

Streaming

How provider events travel from the worker to the caller through the store.

sequenceDiagram
  participant App
  participant C as Caller: ChunkReader
  participant S as Store: P:{r:ID}:chunks
  participant W as Worker: ChunkWriter
  participant P as Provider
  App->>C: converse_stream(...)
  C->>S: call registered and sent (stream=True)
  W->>P: ConverseStream
  loop as events arrive
    P-->>W: event
    W->>S: XADD {seq, att, t: chunk, d} (batched by stream_flush_ms)
    S-->>C: XREAD (one shared read per process)
    C-->>App: yield event
  end
  W->>S: XADD end (usage)
  C-->>App: stream ends

converse_stream / chat_stream start a normal call with stream=True. The worker's ChunkWriter appends entries {seq, att, t: chunk|end|error, d} to P:{r:ID}:chunks: the first event at once, later ones batched by stream_flush_ms (30), capped at stream_max_bytes (8 MiB, then an error, never trimmed); end carries the usage and TPM is corrected after it. Each caller process runs one ChunkReader (XREAD over every open stream, block 100 ms). The caller checks sequence gaps (worker_lost) and idleness (stream_idle_secs). After chunks were sent, a failure ends the stream with an error entry carrying the partial text (CallFailed.partial_text) and no switch to another deployment; a stream already begun by another worker ends worker_lost.