Skip to content

Getting started

This tutorial installs llm-queue, starts a local store and a fake Bedrock, describes two models in a registry and makes a first call. Nothing here calls a real LLM.

You need Python 3.11+ and Docker.

1. Install

pip install "llm-queue[langchain-aws]"            # Bedrock + LangChain
pip install "llm-queue[openai,langchain-openai]"  # OpenAI-compatible servers
pip install "llm-queue[http]"                     # status and health routes for FastAPI

The base install covers Bedrock. Optional features sit behind extras (openai, langchain-aws, langchain-openai, metrics, http); using one without its extra raises MissingExtra, which names the extra to install.

2. Start a local stack

From a clone of the repository:

python scripts/stack.py up        # Valkey, Redis, fakellm (fake Bedrock and OpenAI)
python scripts/stack.py status    # which services answer
Service Port Used for
Valkey 8.1 6379 the store (password localdev)
Redis 7.4 6380 a second store engine for tests
fakellm 8080 a fake Bedrock / OpenAI-compatible server

Any Valkey 8 or Redis 7.2+ works as the store, as one primary (not cluster mode), ideally with maxmemory-policy noeviction. It must allow Lua scripts, pub/sub and blocking list reads; cluster mode is not supported.

3. Write a registry

The registry describes accounts, models and their settings. Save this as registry.json:

{
  "registry_version": 1,
  "residency": {"allowed_regions": ["us-east-1"]},
  "accounts": {"aws-main": {"kind": "aws"}},
  "models": {
    "gpt20b": {
      "type": "chat", "provider": "bedrock", "model_id": "openai.gpt-oss-20b-1:0",
      "rpm": 60, "tpm": 100000,
      "inference": {"max_tokens": 512},
      "deployments": {"aws-main": {"confirmed": ["us-east-1"]}}
    },
    "cohere": {
      "type": "embedding", "provider": "bedrock", "model_id": "cohere.embed-english-v3",
      "rpm": 120, "embed_batch_size": 96,
      "deployments": {"aws-main": {"confirmed": ["us-east-1"]}}
    }
  }
}

Each model has an alias (gpt20b), a provider model id and its budgets. Its deployments name the account and the regions it runs in. Use your own Service Quotas numbers in a real registry.

4. Set the environment

The library reads its process settings from LLMQ_* variables. For the local stack and fakellm:

export LLMQ_ENVIRONMENT="dev"                              # relaxed checks for a laptop
export LLMQ_STORE_URL="redis://:localdev@127.0.0.1:6379/0"
export LLMQ_REGISTRY_FILE="registry.json"
export LLMQ_METRICS_SINK="none"
export LLMQ_BEDROCK_ENDPOINT_URL="http://127.0.0.1:8080"   # fakellm; tests only
export AWS_ACCESS_KEY_ID="aws-main" AWS_SECRET_ACCESS_KEY="fake"

LLMQ_ENVIRONMENT=dev relaxes the production checks. Without it (unset), or with any name other than dev / local / test / ci, the process is strict: LLMQ_STORE_URL must be rediss:// with the certificate checked, and LLMQ_ID_HASH_SALT must be set. Every variable is in Environment variables.

5. Make a first call

import asyncio

from llm_queue import LLMQueue, request_context


async def main() -> None:
    llmq = LLMQueue.from_env()
    await llmq.start()                 # publish / load the registry, run the workers
    try:
        request = {
            "messages": [{"role": "user", "content": [{"text": "Name one stock index."}]}],
            "inferenceConfig": {"maxTokens": 300},
        }
        with request_context(priority=True, session_id="s-1", user_id="u-1"):
            reply = await llmq.converse("gpt20b", request)
        print(reply.response)          # the Bedrock Converse body
        print(reply.meta.deployment)   # "aws-main/us-east-1"

        vectors = (await llmq.embed("cohere", ["first", "second"],
                                    input_type="search_document")).vectors
        print(len(vectors))
    finally:
        await llmq.stop()


asyncio.run(main())

start() publishes the registry to the store (or loads a newer one already there) and runs the worker side in this process. The call is put on the model's queue, admitted by a worker under the model's budgets, and the reply comes back through the store. meta says where and how the call was served.

6. Choose a run mode

LLMQ_MODE decides what start() runs in a process:

Mode Runs Use
both (default) the caller side and the worker side every app pod runs the same code; no separate worker fleet
caller the caller side only processes that make calls but should not run them
worker the worker side and jobs only a separate worker deployment

Budgets, slots, waiter order and health live in the store, so any number of processes in any mode share the same limits.

Next steps

  • Runnable samples of streaming, sync calls, jobs, fallback, status and LangChain: Samples.
  • Every way to call: Make calls.
  • Settings and live edits: Configure.
  • Deploying and running it: Operate.