Errors¶
Everything raised to callers is an LLMQueueError. A failed call raises CallFailed with a
kind; bad configuration raises ConfigError (or one of its subclasses); an admin operation the
store cannot serve raises StoreUnavailable.
LLMQueueError
├── CallFailed a call ended without a reply; branch on .kind
├── ConfigError invalid configuration or a call that can never work; do not retry
│ ├── RegistryError an invalid registry document; .problems lists every problem
│ ├── RegistryConflict a live edit or publish lost the race with other writes every time
│ ├── LiveEditRefused a live edit touched fields that change only with a deploy
│ ├── UnknownModel a model name not in the registry (also a KeyError)
│ └── MissingExtra an optional feature's extra is not installed (also an ImportError)
└── StoreUnavailable the store cannot serve an admin operation (also a ConnectionError)
A plain LLMQueueError is raised for a queue object used where it cannot work: a call from a
forked child of the process that started the queue, an async call from another event loop, a
sync call on the queue's own event-loop thread, or start() on a stopped queue. See
Clients and connections.
CallFailed¶
class CallFailed(LLMQueueError):
kind: Kind # llm_queue.Kind, a Literal of every kind (KINDS as a set)
retry_after_secs: float | None # set where a later retry can work
partial_text: str | None # a stream cut mid-answer
request_id: str | None
| Kind | Meaning | What a caller should do |
|---|---|---|
rate_limited |
No budget before max_wait_secs or the deadline, the provider kept throttling past throttle_retry_attempts (or the retry budget ran out with no untried deployment left), or a fallback model was busy |
Retry after retry_after_secs, or tell the user to wait |
queue_full |
The model's queue already holds max_waiting (50,000) calls, or send_job found 10,000 jobs of that name waiting; retry_after_secs is 5 |
Retry later; shed load |
deadline_exceeded |
No reply within the call's timeout (also: the store did not answer in time, or retries ran out of time) | Retry once, or raise timeout_secs for long answers |
invalid_request |
The request itself is wrong: an embedding text is empty, the provider refused the request as invalid, a max_tokens / max_completion_tokens is not a whole number on a model with a cap, or on_exceed: reject refused its inference values |
Fix the request; retrying will not help |
context_length_exceeded |
The input does not fit the model's context window | Shorten the input |
request_too_large |
The call is too big: its input estimate is above every candidate deployment's tpm, or its message is above the model's max_message_bytes (4 MB) |
Shorten the input or split it, or raise max_message_bytes; retrying will not help |
no_deployment |
The call's own filters (allowed_regions, allowed_accounts) leave no deployment |
Fix the filters or the registry |
model_unavailable |
No usable deployment: all disabled, blocked, unhealthy or outside residency; retry_after_secs is the time to the next probe |
Use another feature or model, or retry after retry_after_secs |
model_disabled |
The model is disabled and has no fallback model to use | Use another model; an operator turned it off |
queue_unavailable |
The store is unreachable, read-only, out of memory, or the model's admission numbers are missing (store_down: fail_fast, or pause that ran out of time); a stream needs the store |
Retry later; check the store |
credentials_unavailable |
Only accounts whose credentials were refused are left; retry_after_secs is 60 |
Check the account's credentials / role; retry later |
retry_budget_exhausted |
Provider errors on every deployment, and the model's retry budget is spent | Retry later; the provider is failing |
provider_error |
A fatal provider error, or max_attempts reached (also an embedding reply with no body) |
Show an error; check the request if it repeats |
request_id_conflict |
Your request_id was used before with other params or another user |
Use a new id |
result_expired |
Your request_id finished earlier and its result is gone, or a result offloaded to S3 could not be read back |
Use a new id, or keep results longer (LLMQ_RESULT_TTL_SECS) |
dead_lettered |
The call failed max_receives (10) times (workers died holding it) and went to the dead-letter queue |
Report it; see llm-queue dlq list |
worker_lost |
A stream ended without its end entry or had a gap (its worker was lost) | Retry; partial_text holds the text so far |
worker_shutdown |
A stream had already sent chunks when its worker's drain ran out (a call that has sent nothing is given back to the queue instead and does not fail) | Retry; partial_text holds the text so far |
cancelled |
The call was cancelled (cancel(), or its caller process was gone) |
Nothing, unless you did not cancel it |
rejected |
The app's admission hook rejected the call before it was sent, or the hook raised, or the registry names one this process has not registered (fails closed) |
Handle as your hook intends; for a failing hook, check llmq_hook_failed_total |
The set of kinds is llm_queue.KINDS. Over the OpenAI-compatible HTTP transport the LangChain
OpenAI adapters use, kinds map to status codes. Only the request's own faults are 4xx; failures
after the queue gave up are 5xx (with retry-after when known), so the SDK and LangChain treat
them as server-side, not as a bad request:
| Status | Kinds |
|---|---|
| 400 | invalid_request, context_length_exceeded, request_too_large |
| 403 / 404 / 409 / 410 / 499 | rejected / model_disabled / request_id_conflict / result_expired / cancelled |
| 429 | rate_limited, queue_full |
| 502 | provider_error, worker_lost, dead_lettered |
| 503 | retry_budget_exhausted, no_deployment, queue_unavailable, model_unavailable, credentials_unavailable, worker_shutdown |
| 504 | deadline_exceeded |
| 500 | any other kind |
StoreUnavailable¶
Raised by cancel, status, request_status, send_job and the registry's reads and edits
when the state store is unreachable, read-only (after a failover) or full. Calls never raise it;
they raise CallFailed (queue_unavailable). It is also a ConnectionError. Retry after the
store is back; see Operate.
ConfigError¶
Invalid configuration, or a call that can never work as written. Fix the configuration or the call; retrying will not help. The message lists every problem at once. It is raised:
- by
from_env: badLLMQ_*settings orTuning, an unreadable registry file or document; - by
start(): an invalid registry, a store in cluster mode; - by an admin action: a refused registry edit or publish,
http_router(allow_writes=True)withoutdependencies, an empty or malformed auditactor; - by a call, before anything is queued: a model name not in the registry (
UnknownModel), a streaming call to an embedding model, a feature whose extra is missing (MissingExtra).
A call that was accepted never ends in a ConfigError; whatever goes wrong after that is a
CallFailed.
Subclasses:
RegistryError: an invalid registry document;.problemsand.warningslist each one with its path. A live edit is checked strictly on the paths it adds or changes, so a typo such asrmp=400is refused instead of being stored and ignored.RegistryConflict: a live edit (or a publish) raced other writers and lost every retry (RegistryOptions.edit_retries, 3). Nothing was written; run it again.LiveEditRefused: a live edit changed fields the live-edit mode does not allow (see Live edits), or live edits areoff. The message lists each refused path and why.UnknownModel: a model name that is neither an alias nor a model id in the registry. Also aKeyError(the HTTP router answers 404).MissingExtra: an optional feature's dependencies are not installed. The message names the extra:pip install 'llm-queue[<extra>]'. Also anImportError.