06 Jobs¶
Queued background work that runs at least once, with LLM calls inside it. Use it for ingestion, nightly re-embedding, and any batch work that should not block a request.
sequenceDiagram
participant Any as Any process
participant S as Valkey
participant W as Worker (registered the job)
participant B as Bedrock
Any->>S: send_job("index_documents", payload)
S-->>W: pick job
W->>S: embed("cohere", texts): admit + enqueue (normal priority)
S-->>W: pick call
W->>B: InvokeModel
B-->>W: vectors
W->>S: ack job (or retry if it raised)
Key points¶
- Register the job function in every worker process before
start(). Any process can send a job. - A job that raises is retried, so make it safe to run twice (for example, upsert by id).
- LLM calls inside a job go through the same queue and budgets as user calls, at normal priority.
Code¶
samples/06_jobs.py
"""Background jobs (ingestion, nightly re-embedding): queued work that runs at least once.
Register the job function in every worker process before `start()`; any process can send a
job. A job that raises is retried, so make it safe to run twice (upsert by
`current_job().job_id`). LLM calls made inside a job go through the same queue and budgets as
user calls, at normal priority. Jobs return nothing: this sample learns "done" through an
in-process `asyncio.Queue`, which works only because sender and worker are one process.
python scripts/samples.py 06_jobs [--live]
"""
from __future__ import annotations
import asyncio
from typing import Any
from llm_queue import LLMQueue
done: asyncio.Queue[dict[str, Any]] = asyncio.Queue()
async def main() -> None:
llmq = LLMQueue.from_env()
async def index_documents(payload: dict[str, Any]) -> None:
result = await llmq.embed("cohere", payload["texts"], input_type="search_document")
# write result.vectors to the vector store here, keyed by payload["ids"]
await done.put({"ids": payload["ids"], "vectors": len(result.vectors)})
llmq.register_job("index_documents", index_documents)
await llmq.start()
try:
job_id = await llmq.send_job(
"index_documents", {"ids": ["d1", "d2"], "texts": ["first report", "second report"]}
)
print(f"sent job {job_id}")
finished = await asyncio.wait_for(done.get(), timeout=60)
print(f"job finished: {finished}")
assert finished["vectors"] == 2
finally:
await llmq.stop()
if __name__ == "__main__":
asyncio.run(main())
Output (live, 2026-11-14)¶
See also: Background tasks.