Run background jobs¶
A job is work that should not block a request: ingestion, re-embedding, a nightly report. You register an async function under a name, send it a JSON payload, and a worker process runs it.
from llm_queue import current_job
async def index_documents(payload: dict) -> None:
job = current_job() # name, job_id, attempt, max_attempts
result = await llmq.embed("cohere", payload["texts"], input_type="search_document")
await save_vectors(job.job_id, payload["ids"], result.vectors) # upsert by job id
llmq.register_job("index_documents", index_documents, timeout_secs=600) # before start()
await llmq.start()
job_id = await llmq.send_job("index_documents", {"ids": ids, "texts": texts}, job_id=batch_id)
Guarantees and limits¶
| Delivery | At least once. A job can run twice (a worker killed mid-job, an ack lost, a send retried). Make it idempotent: write by current_job().job_id |
| Result | Fire and forget. The return value is dropped; there is no job status (request_status(job_id) is empty). Write the outcome where the app can read it |
| Where it runs | Processes in both or worker mode that registered the name. register_job in caller mode, or after start(), raises ConfigError |
| The function | Must be async def (or a partial of one, or an object with an async __call__): a plain def is refused at register_job. Never block the event loop: put blocking SDK calls and CPU work in asyncio.to_thread, or the job's heartbeat stops and the job is handed to another worker |
| Names | 1-64 letters, digits, _ or - |
| Payload | JSON only, at most 1 MB (LLMQ_JOB_MAX_PAYLOAD_BYTES). A larger one fails request_too_large; one orjson cannot write (a Decimal, a set) fails invalid_request. Tuples arrive as lists, datetime and UUID as strings. Keep large inputs in your own storage and send a reference |
| Dedupe | send_job(job_id=X) again within 24 h (LLMQ_JOB_DEDUPE_SECS) queues nothing and returns X. Without job_id every send is a new job |
| Order | Not kept: up to 4 jobs of a name run at once per process, and a retried job goes back in later |
| Concurrency | 4 per job name per process (register_job(..., concurrency=N) or LLMQ_JOB_CONCURRENCY), 32 per process over all names (LLMQ_JOB_PROCESS_CONCURRENCY) |
| Waiting | At most 10,000 waiting jobs per name (CallFailed("queue_full"), retry_after_secs 5); a job waits at most the queue retention (1 day) before it is trimmed |
| Durability | With LLMQ_QUEUE_WAIT_REPLICAS (scope jobs in LLMQ_QUEUE_WAIT_SCOPES), a send waits for the replicas like a call does; with too few it is sent anyway, warned (replicas_short) and counted |
| Context | A job's LLM calls run at normal priority with the sender's user_id and tenant_id (from request_context at send_job), not with the context of the process that called start() |
| LLM budget | Jobs take none of their own; their LLM calls do |
Failures, retries and dead jobs¶
A job that raises is retried after 5 s, then 10, 20, 40 ... up to 5 minutes between tries, for
about an hour in total (LLMQ_JOB_RETRY_BASE_SECS, LLMQ_JOB_RETRY_MAX_SECS,
LLMQ_JOB_RETRY_FOR_SECS); a CallFailed with retry_after_secs waits at least that long. Then
it is dead: moved to the job's dead letters, kept 7 days (LLMQ_JOB_DEAD_RETENTION_SECS),
logged job_failed outcome=dead at error level with the traceback, and counted in
llmq_jobs_total{outcome="dead"}.
timeout_secs: a run longer than this is cancelled and counts as a failure (logged withkind=job_timeout). Without it a hung job holds its worker slot for good. It is the limit of the process that runs the job: during a rolling deploy that changes it, old and new pods use their own values.- Shutdown: a job still running when the drain ends is returned to the queue, not failed.
A job returned 5 times (
LLMQ_JOB_MAX_RETURNS) is dead-lettered: a job longer than the drain otherwise restarted from zero at every deploy. Checkpoint long jobs (save progress under the job id and skip what is done). - No consumer:
send_jobto a name no running process consumes still queues the job, but logsjob_no_consumer(once a minute per name) and countsllmq_jobs_no_consumer_total.
Dead jobs:
dead = await llmq.job_dead_letters("index_documents") # id, job_id, fails, payload
await llmq.redrive_jobs("index_documents") # run them again
llm-queue dlq list --job index_documents
llm-queue dlq redrive --job index_documents --yes
llm-queue queues list # job queues included
status() lists every job name a worker consumed lately (jobs: waiting and dead letters), and
the job_dead_letters alert turns on while a job name has any dead job.