05 Errors and fallback¶
Handle failures with one exception type, CallFailed, and a kind to branch on. Every caller
needs this: map kind to a user message or a retry.
flowchart LR
c1["converse('gpt120b')"] --> d1{"gpt120b<br/>deployment on?"}
d1 -- no --> fb["model_fallback<br/>→ gpt20b"]
fb --> ok["reply<br/>meta.fallback = 'model'"]
c2["converse('gpt20b')"] --> d2{"gpt20b<br/>deployment on?"}
d2 -- "no, no fallback" --> err["CallFailed<br/>kind = 'model_unavailable'"]
Key points¶
- The sample turns deployments off with a live registry edit (what
llm-queue registry disabledoes), and turns them back on at the end. - A model with a
model_fallbackrule is served by its fallback model, andmeta.fallbacksays so. - A model with nowhere to go fails fast with
CallFailed.retry_after_secstells you when to try again. - Branch on
kind:rate_limitedandqueue_fullmean busy: retry later.model_unavailable,model_disabledandno_deploymentmean nothing can serve the model now.
Code¶
samples/05_errors_and_fallback.py
"""Handle failures: one exception type (`CallFailed`) with a `kind` to branch on.
Turns a deployment off with a live registry edit (what `llm-queue registry disable` does) to
show two things: a model with a fallback model is served by it (`meta.fallback == "model"`),
and a model with nowhere to go fails fast with `CallFailed`. Both edits are undone at the end.
python scripts/samples.py 05_errors_and_fallback [--live]
"""
from __future__ import annotations
import asyncio
from llm_queue import CallFailed, LLMQueue
REQUEST = {
"messages": [{"role": "user", "content": [{"text": "Say hi."}]}],
"inferenceConfig": {"maxTokens": 200},
}
async def main() -> None:
llmq = LLMQueue.from_env()
await llmq.start()
view = llmq.registry.current
big = view.deployments("gpt120b")[0].key # e.g. "aws-main/us-east-1"
small = view.deployments("gpt20b")[0].key
try:
# 1. gpt120b is off: its `model_fallback` sends the call to gpt20b.
await llmq.registry.set_enabled("gpt120b", big, False, actor="sample")
reply = await llmq.converse("gpt120b", REQUEST, max_wait_secs=30)
print(
f"gpt120b off -> served by {reply.meta.served_model} (fallback={reply.meta.fallback})"
)
assert reply.meta.served_model == "gpt20b"
# 2. gpt20b off as well: nothing can serve it, the call fails with a kind.
await llmq.registry.set_enabled("gpt20b", small, False, actor="sample")
try:
await llmq.converse("gpt20b", REQUEST, max_wait_secs=3)
except CallFailed as exc:
print(f"gpt20b off -> CallFailed kind={exc.kind!r} retry_after={exc.retry_after_secs}")
if exc.kind in ("rate_limited", "queue_full"):
pass # busy: retry after exc.retry_after_secs, or tell the user to wait
elif exc.kind in ("model_unavailable", "model_disabled", "no_deployment"):
pass # nothing can serve the model now: show an error or use another feature
else:
raise AssertionError("expected CallFailed")
finally:
await llmq.registry.set_enabled("gpt120b", big, True, actor="sample")
await llmq.registry.set_enabled("gpt20b", small, True, actor="sample")
await llmq.stop()
if __name__ == "__main__":
asyncio.run(main())
Output (live, 2026-11-14)¶
gpt120b off -> served by gpt20b (fallback=model)
gpt20b off -> CallFailed kind='model_unavailable' retry_after=60.0
See also: Errors, Failures and recovery.