Skip to content

Performance (measured)

Measured numbers, with where each comes from. Nothing here is an estimate unless it says so.

flowchart LR
  subgraph caller["Caller"]
    c1["REGISTER"] --> c2["SEND + doorbell"] --> c3["CONFIRM"]
    c4["RESULT_TAKE"]
  end
  subgraph worker["Worker"]
    w1["RECEIVE (batched)"] --> w2["CALLERS_ALIVE"] --> w3["PICK_TAKE<br/>p99 0.94 ms, 600 deployments"]
    w3 --> w4["provider call"] --> w5["FINISH"] --> w6["RESULT_PUT + PUBLISH"] --> w7["STATUS_SET, ACK"]
  end
  c3 -. "wake: one doorbell token" .-> w1
  w6 -. "wake: pub/sub" .-> c4
  note["11.6 store scripts per call (B19)<br/>+0-15 ms p95 over a direct call (B34)<br/>290 calls/s on 4 workers"]

Every step above is one store round trip; the measured numbers below say what each costs.

Numbers from the CHANGELOG "Performance" section and plan ยง13 (maintainers' notes in docs/internal, not published) (2026-11-13, Python 3.12, fakellm, worker processes as pods on one machine).

What Measured
Store scripts per queued call 11.6 (B19, 2,000 calls in 10 s). By the code: caller REGISTER, SEND, CONFIRM, RESULT_TAKE; worker RECEIVE (batched), CALLERS_ALIVE, PICK_TAKE, first-attempt counter, FINISH, RESULT_PUT, STATUS_SET, ACK; plus doorbell and heartbeat. About 4 ms of store time per call on an in-VPC store (plan estimate)
Store work vs process count 248 commands per call on 2 processes, 252 on 8 (B22)
Admission PICK_TAKE p99 0.94 ms with 20 models x 30 deployments (B91; plan asks 2 ms)
Status 20 x 30 report (210 KB) in 47 ms, 375 ms cold; 6.5 s before one script per model (B91)
Queue overhead +0 ms at p95 over a direct call, 141 ms both (B34); CHANGELOG: 0-15 ms p95; 290 calls/s on 4 workers
Waiting callers 2,000 waiting: 1,976 store calls/s before, 499/s after (pub/sub + sweep, B56)
Budgets 1,800 of 1,800 a minute on 9 deployments, none above its RPM, priority wait p95 0.27 s (B17); normal class at 100 % of its 80 % (B56 / B93)
Recovery Waiter pod killed: next waiter within 1.3 s smooth, 0.02 s after the minute fixed (B94); first call after idle 125 ms (B92)
Memory Bulk Cohere embedding (300 x 96 texts): store +23 MB at peak, +582 MB before deleting results on first read (B97; CHANGELOG states 31 MB vs 587 MB). Process memory: unverified
CPU per call botocore parameter validation off: 8.5 to 3.5 ms per attempt at 218 KB; no deep copy (1.86 ms saved); envelope encode 342 to 57 us; UUIDv7 0.49 us; hiredis parsing 39-75x faster
Message size check One orjson pass for max_message_bytes: 0.22 ms at 240 KB (2026-11-14, Python 3.12)

Client start-up and per-call overhead: 0.1.0b1 baseline

scripts/bench.py measures pod start, the cold first call, the first call after an idle spell, steady overhead and stream overhead on the local stack (fakellm with a fixed 5 ms latency), using only the public API, so every release is measured the same way. The 0.1.0b1 run is the "before" for the 0.1.0b2 client hub, warm connections and round-trip folding (scripts/baselines/bench-0.1.0b1.json, 2026-11-14, Python 3.12, Windows, 6 deployments).

What (p50 / p95, ms) 0.1.0b1
Pod start (from_env to start()) 145 / 336
Cold first call 3,010 / 3,435
First call after 60 s idle 360 / 640
Queue overhead, sequential priority calls 125 / 425
Queue overhead, 20 concurrent 774 / 1,593
Stream overhead to the first event 123 / 366
Store round trips per call, converse / converse_stream (median) 18.6 / 30.9

Indicative

Other test processes may have been running on the host; compare only runs made back to back on the same machine.

0.1.0b2 was released without a back-to-back benchmark against 0.1.0b1; the measured before / after numbers of each change are in the CHANGELOG's Performance section.