Skip to content

06 Jobs

Queued background work that runs at least once, with LLM calls inside it. Use it for ingestion, nightly re-embedding, and any batch work that should not block a request.

sequenceDiagram
  participant Any as Any process
  participant S as Valkey
  participant W as Worker (registered the job)
  participant B as Bedrock
  Any->>S: send_job("index_documents", payload)
  S-->>W: pick job
  W->>S: embed("cohere", texts): admit + enqueue (normal priority)
  S-->>W: pick call
  W->>B: InvokeModel
  B-->>W: vectors
  W->>S: ack job (or retry if it raised)

Key points

  • Register the job function in every worker process before start(). Any process can send a job.
  • A job that raises is retried, so make it safe to run twice (for example, upsert by id).
  • LLM calls inside a job go through the same queue and budgets as user calls, at normal priority.

Code

samples/06_jobs.py
"""Background jobs (ingestion, nightly re-embedding): queued work that runs at least once.

Register the job function in every worker process before `start()`; any process can send a
job. A job that raises is retried, so make it safe to run twice (upsert by
`current_job().job_id`). LLM calls made inside a job go through the same queue and budgets as
user calls, at normal priority. Jobs return nothing: this sample learns "done" through an
in-process `asyncio.Queue`, which works only because sender and worker are one process.

    python scripts/samples.py 06_jobs [--live]
"""

from __future__ import annotations

import asyncio
from typing import Any

from llm_queue import LLMQueue

done: asyncio.Queue[dict[str, Any]] = asyncio.Queue()


async def main() -> None:
    llmq = LLMQueue.from_env()

    async def index_documents(payload: dict[str, Any]) -> None:
        result = await llmq.embed("cohere", payload["texts"], input_type="search_document")
        # write result.vectors to the vector store here, keyed by payload["ids"]
        await done.put({"ids": payload["ids"], "vectors": len(result.vectors)})

    llmq.register_job("index_documents", index_documents)
    await llmq.start()
    try:
        job_id = await llmq.send_job(
            "index_documents", {"ids": ["d1", "d2"], "texts": ["first report", "second report"]}
        )
        print(f"sent job {job_id}")
        finished = await asyncio.wait_for(done.get(), timeout=60)
        print(f"job finished: {finished}")
        assert finished["vectors"] == 2
    finally:
        await llmq.stop()


if __name__ == "__main__":
    asyncio.run(main())

Output (live, 2026-11-14)

sent job 01a1d83e8f6f70ad9f39f456071a3ecd
job finished: {'ids': ['d1', 'd2'], 'vectors': 2}

See also: Background tasks.