Skip to content

Admission

How a worker decides, in one atomic store script, whether a call may run now, where, and what it costs.

flowchart TD
  A([PICK_TAKE: one script, all or nothing]) --> B{Config current?}
  B -- "changed" --> B1([stale_config: reload, ask again])
  B -- yes --> C{Already settled?}
  C -- "result or final state" --> C1([done])
  C -- "cancel flag" --> C2([cancelled])
  C -- "running elsewhere" --> C3([duplicate])
  C -- no --> D[Reserve for older waiters<br/>and for waiting priority calls]
  D --> E[Each candidate: enabled, residency,<br/>health, breaker]
  E --> F{Any usable?}
  F -- "no, a breaker probe in flight" --> F1([wait: probe])
  F -- no --> F2([none: fallback or model_unavailable])
  F -- yes --> G[Budgets per candidate: RPM, TPM,<br/>class share, account budget, slots]
  G --> H{Fits?}
  H -- yes --> I[Choose by strategy:<br/>ordered, round_robin, weighted, headroom, custom]
  I --> J([granted: write budgets, slot lease,<br/>in-progress entry, claim])
  H -- no --> K([wait: register as waiter,<br/>shortest wait and its reason])

One loop per model. One script (RECEIVE_FIRST) reads the priority queue first, then the normal one, up to min(free worker_slots, 10) messages, and returns the store's time, so queue_wait_ms is measured on the store clock (RECEIVE semantics: due returned messages and new ones, oldest first; each taken message is hidden for queue.visibility_secs, its generation goes up; a hold that lapsed counts one failure and the tenth goes to the dead-letter stream). Idle, the loop blocks on both doorbells (BLPOP) for 50 ms, then for 1 s after 20 empty polls; a send wakes exactly one blocked worker. A worker woken with no free slot, or for a class it is holding, pushes the token back so another pod wakes instead.

For each message: parse (a newer envelope version is returned after 15 s with no failure counted, never dropped); then admission. PICK_TAKE itself checks, on a call's first admission, that a caller is alive (dead callers mean caller_gone), counts the first attempt for the retry budget, and reports a stream call's chunk-stream length, so none of these is a separate round trip.

PICK_TAKE (one script, all or nothing) does, in order:

Step What happens
Config Reads the model's resolved numbers from P:{m:A}:cfg (written with every publish / edit). A change counter different from the worker's means stale_config (the worker reloads and asks again); missing numbers mean no_config (republish, then queue_unavailable or return in 5 s)
Settled? A stored result or a final job state: done; a cancel flag: cancelled; running under another worker with a fresh heartbeat: duplicate (message returned after half a lease). A dead owner's entry and slot are dropped
Waiters Expired waiter entries are cleaned. A foreign (model-fallback) ask is refused busy while the model has waiters of its own
Tenant Only with fairness on (Tenant fairness): quota refuses a tenant past its own rpm / tpm (tenant_limited: the call fails rate_limited); share makes a tenant over its weighted share wait (tenant) while a tenant within its share waits; fair_queue ranks the call among the waiters by its tenant round instead of its queue position
Reservation Older waiters of the same class set aside their cost (1 request, their tokens, 1 slot) on every deployment they may use. A normal call also sets aside what waiting priority calls need, on a deployment whose normal use this minute reached normal_min_share x rpm
Candidates Each passed candidate is usable unless disabled, its account disabled, blocked, outside residency, auto-disabled (health), or its breaker open (after the cooldown one trial call at a time)
Budgets Per candidate: the deployment's rpm / tpm, the class share (normal at most 1 - priority_reserve, priority at most 1 - normal_min_share), and the account's shared_rpm / shared_tpm (with the account's own priority_reserve for normal calls). smooth (default): a GCRA clock per budget, cost 60 / limit s per unit, burst max(1, floor(limit x rate_burst)). fixed: counters per calendar minute. A call larger than the whole bucket is admitted when the clock is idle
Slots Concurrency per class: limit from P:{m:A}:ac:DEP (starts at max_concurrency, 200 when unset; x0.7 on a throttle at most once a second, +1/limit per success up to the cap); normal slots at most (1 - priority_reserve) of it, priority at most (1 - normal_min_share)
Slow With 3+ candidates that have slow_min_samples (20) latency samples, one above slow_ratio (3) x the median and above slow_min_secs (5) becomes a last choice
Choose ordered (default) stops at the first fitting confirmed candidate; round_robin, weighted (smooth weighted round robin), headroom (pick: percent = largest free share of the bucket / minute and slots, before the call; max; spread), custom:<name> (the hook orders, then ordered). Order of preference: confirmed, then fallback, then last choices (previously failed or slow). A sticky session keeps its deployment while it fits or would wait at most sticky_move_after_secs (1 s)
Headroom rule A foreign ask with require_headroom needs the chosen deployment's free share at or above require_headroom_pct (20 %)
Take or wait Granted: write the budget states and per-minute counters, the slot lease, P:{m:A}:inprog and inprog:ID (owner, deployment, class, minute, cost), the claim (job state=running, owner, hb), the sticky key, the rotation; remove the waiter. Refused: register or refresh the waiter (wait:p / wait:n scored by queue position, waitinfo with cost and usable deployments, expiry 2 s) and return wait with the shortest wait and its reason (rpm, tpm, account, reserve, not_head, slot; tenant before any deployment is tried, with fairness share). No usable candidate at all: none

Python then handles the terminal outcomes: an input estimate above every candidate's tpm fails request_too_large; none turns on fallback regions for the cause broken (after fallback.after_secs) or tries fallback models, else model_unavailable with retry_after_secs until the next probe; no candidate after the call's filters is no_deployment, or credentials_unavailable when only accounts with failed credentials were left.

Hold. A refused first attempt is held: the class stops being read (the calls behind it stay in the queue, untouched), and a task retries PICK_TAKE every min(wait, 1 s), extending the message's visibility each time. Fallback regions join the candidates once the call has waited fallback.after_secs (2) and the cause (exhausted, broken, error) is listed; fallback models are tried after model_fallback.after_secs (5), with the other model's own budgets and estimate, only while it has no waiters. The fallback_policy hook can override both. Past max_wait_secs (counted from the queue position) or the deadline the call ends expired, and the caller gets rate_limited with retry_after_secs. A call that already waited out max_wait_secs unreceived is refused on arrival rather than started with no time left. When the hold ends the loop is woken.