Skip to content

07 Status and registry

Check the status of models and calls, and edit the registry live. Use it for admin pages, runbooks and readiness checks.

Key points

  • status(model) shows waiting calls, calls in progress, and each deployment's RPM / TPM used and left this minute, its health and its breaker.
  • request_status(id) follows a single call.
  • registry.set changes a safe setting live in every pod. Each change is audited, and registry.log lists the changes.
  • The same views are on the CLI (llm-queue) and the HTTP router.

Code

samples/07_status_and_registry.py
"""Operate: status of models and calls, and live registry edits (admin pages, runbooks).

`status(model)` shows waiting calls, calls in progress and each deployment's RPM / TPM used
and left this minute. `request_status(id)` follows one call. `registry.set` changes a safe
setting live in every pod (audited); `registry.log` lists the changes. The CLI `llm-queue`
and the HTTP router (`llmq.http_router()`, extra `[http]`) expose the same.

    python scripts/samples.py 07_status_and_registry [--live]
"""

from __future__ import annotations

import asyncio

from llm_queue import LLMQueue

REQUEST = {
    "messages": [{"role": "user", "content": [{"text": "One word: ok"}]}],
    "inferenceConfig": {"maxTokens": 100},
}


async def main() -> None:
    llmq = LLMQueue.from_env()
    await llmq.start()
    try:
        reply = await llmq.converse("gpt20b", REQUEST)
        call = await llmq.request_status(reply.request_id)
        print(f"request {reply.request_id}: state={call['state']}")

        model = await llmq.status("gpt20b")
        for dep in model["deployments"]:
            print(
                f"{dep['name']}: rpm {dep['rpm']['used']}/{dep['rpm']['limit']} "
                f"tpm {dep['tpm']['used']}/{dep['tpm']['limit']} healthy={dep['healthy']} "
                f"breaker={dep['breaker']}"
            )
        print(f"waiting={model['waiting']} in_progress={model['in_progress']}")

        before = llmq.registry.current.value("gpt20b", "rpm")
        await llmq.registry.set({"rpm": 45}, model="gpt20b", actor="sample")
        print(f"gpt20b rpm {before} -> {llmq.registry.current.value('gpt20b', 'rpm')} (live)")
        for entry in await llmq.registry.log(count=3):
            print(f"  audit: {entry}")
        await llmq.registry.set({"rpm": before}, model="gpt20b", actor="sample")
        assert call["state"] == "done"
        print(f"health: {await llmq.health()}")
    finally:
        await llmq.stop()


if __name__ == "__main__":
    asyncio.run(main())

Output (live, 2026-11-14)

request 01a1d83ea22470e1acb8721297d2277b: state=done
aws-main/us-east-1: rpm 1/60 tpm 143/100000 healthy=True breaker=closed
waiting={'priority': 0, 'normal': 0} in_progress=0
gpt20b rpm 60 -> 45 (live)
  audit: {'actor': 'sample', 'op': 'edit', 'diff': [{'path': 'models.gpt20b.rpm', 'before': 60, 'after': 45}], ...}
  audit: {'actor': 'sample', 'op': 'edit', 'diff': [{'path': 'models.gpt20b.deployments.aws-main.overrides.us-east-1.enabled', 'before': False, 'after': None}], ...}
  audit: {'actor': 'sample', 'op': 'edit', 'diff': [{'path': 'models.gpt120b.deployments.aws-main.overrides.us-east-1.enabled', 'before': False, 'after': None}], ...}
health: {'live': True, 'ready': True, 'stores': {'state': 'ok', 'queue': 'ok'}}

The last two audit entries come from sample 05 turning its deployments back on. The registry and its audit log are shared by every process using the same prefix.

See also: Operate, CLI and HTTP.