Skip to content

Contributing to llm-queue

This is a library: other apps install it and run it inside their own processes. Most of the rules below follow from that. Where this file and the house standard disagree, this file wins.

Setup

You need Python 3.12+, uv and Docker.

uv sync --all-extras          # .venv with every extra and the dev group (uv.lock)
python scripts/stack.py up    # or: make up
python scripts/stack.py status
Service Port Used for
Valkey 8.1 6379 the main store for stack tests, samples, drills
Redis 7.4 6380 the second store engine; every engine test runs on both
fakellm 8080 a fake Bedrock / OpenAI-compatible server (tools/fakellm)

python scripts/stack.py rebuild after changing tools/fakellm; down to stop. The store password for tests is localdev (LLMQ_TEST_REDIS_PASSWORD overrides it).

uv.lock is the lock file. Change dependencies in pyproject.toml and run uv lock; never edit the lock by hand, and commit both together. Dependencies are ranges (>=x,<y), never pins, and each one has a comment saying why it is there and why the bound is where it is. Optional features go behind an extra (openai, langchain-aws, langchain-openai, metrics, http), not into the base install.

Running checks

Every check goes through scripts/check.py. A step is done only when its check passes.

python scripts/check.py --list      # the steps and what each one runs
python scripts/check.py a6          # one step
python scripts/check.py a6 a8       # several
python scripts/check.py dev         # dev run (= all --quick): scenarios at 100 % only,
                                    # gates without B17 / B91 / B92
python scripts/check.py all         # before a push (or: make check-all): adds the burst
                                    # scenarios, every gate and coverage
python scripts/check.py all --release  # before a release: every scenario x every load
python scripts/check.py soak3       # 3-minute soak: smoke test while fixing
python scripts/check.py soak10      # 10 minutes: after a change to the engine
python scripts/check.py soak20      # 20 minutes: before a release

Logs. Every check writes its whole output, live, to reports/check-<time>/<step>.<check>.log (git-ignored); the console prints one PASS / FAIL line per check with its log path, and summary.log in the same folder keeps the console. pytest runs with -rA, so a log lists every test and its outcome. Follow a long drill with tail -f reports/check-<time>/gates.*.log.

Scenarios first. scripts/scenarios.py runs one fault at a time (or a chosen pair) and judges it by calls, not by time: a phase of healthy calls, the fault with calls still coming until it ends, then a phase after it settles. Each scenario has its own pass rule and runs under three loads (50 %, 100 % and 300 % of the budget). The fast profile (default) shortens leases, visibility, deadlines and lifetimes, so a scenario takes 10-40 s; --profile prod re-runs it with production timers.

python scripts/scenarios.py --list           # every scenario and its rule
python scripts/scenarios.py S4 --load under  # one, while fixing it
python scripts/scenarios.py --group store    # a group
python scripts/check.py scenarios --quick    # every scenario, one load each
Group Scenarios
worker S1 killed, S2 graceful restart, S3 all down then back
store S4 blocked, S5 half-open, S8 keys reset, S9 out of memory, S10 slow; S6 / S7 / S11 failover drills on Docker nodes
provider (fakellm) P1 throttled, P2 5xx, P3 hangs, P4 model not in region, P5 bad credentials, P7 every region down
registry R1 rpm cut live, R2 region disabled, R3 kill switch, R4 bad edit refused
caller C1 cancel, C2 caller process dies, C3 duplicate request id
pairs X1-X8: two faults at once (store block + worker kill, rpm cut + throttle ...)

A new bug gets a scenario (or a check in one) before its fix, like a unit test.

Short first, longer before a release. all --quick leaves out the three slow calibration gates. The soak has three lengths, and each scales its chaos spacing, store block, status expiry and sampling to fit, so even 3 minutes sees every chaos event:

Step Calls Chaos Use
soak3 ~12,600 every 30 s, block 10 s smoke test while fixing
soak10 ~42,000 every 60 s, block 20 s after a change to the engine
soak20 ~84,000 every 120 s, block 20 s before a release

Shorter runs catch broken recovery, leftovers and deadline bugs as well as long ones; only time shows slow leaks and drift (more calls, more growth above the noise). Run all --release and soak20 before a release.

Each step runs ruff check, ruff format --check and mypy on its source paths, then pytest on its tests, then any extra script. all instead lints and type-checks the union of every step's paths once, runs the unit tests once and the stack tests once, then each step's extra script; each check has a time limit, so a hung drill fails instead of hanging. Steps that need the stack fail at once if a service does not answer; they never skip silently.

Steps What they cover
a0 fakellm itself (accounts, TPM, speed, hangs, streaming)
tools test tooling: the provider error table, budget property tests, 200-way contention
a1-a2 registry: schema, settings, validation; store, live edits, audit, rollback
a3 clients per account and region, providers, attempt timers
a4-a6 keys, message format, stream queue, admission (Lua)
a7-a13 tokens, engine end to end, hooks, operations, health, status, embeddings
a14-a17 front door, LangChain adapters, assume-role, latency exclusion, streaming
store store failover tests plus scripts/failover.py (Docker failover drill)
load scripts/load.py: 2,000 calls over three workers; a burst over the RPM budget
gates scripts/gates.py: the plan's load and chaos gates on worker processes (~20 min)
soak3 / soak10 / soak20 (not in all) scripts/soak.py --minutes N: 8 worker processes, chaos (kill, store block, live edit, restart), leak and drift checks; soak20 before a release
samples scripts/samples.py: every file in samples/ run as app code on fakellm
bench (not gating, not in all) scripts/bench.py: prints its numbers; a failure never fails the run

No test calls a real LLM. Unit tests use fakes, stack tests and drills use fakellm. The only path to a real model is python scripts/samples.py --live (real Bedrock, AWS keys read from --env-file, default .env), and it runs only when someone asks for it. --store env points the samples at the env file's store; the run touches only its own key prefix and deletes it afterwards.

CI (.github/workflows/ci.yml) runs ruff, format, mypy and the unit tests on Python 3.12-3.14, the unit tests on Windows and on the lowest allowed versions (--resolution lowest-direct), the LangChain 0.3 adapters, and unit plus integration tests against the same Docker stack on Python 3.12, 3.13 and 3.14 (with coverage on 3.12). The nightly workflow (.github/workflows/nightly.yml, 02:17 UTC or by hand) runs the slow drills CI cannot run on every push: scenarios.py all --load under, gates.py --quick and a 10-minute soak, all on fakellm, and keeps reports/ for 7 days.

Adding a check step

Add an entry to STEPS in scripts/check.py: a title, the sources it lints and type-checks, the tests it runs (files or file::test ids), needs_stack=True if it uses Docker, and extra for a script to run after the tests. Keep the source paths narrow so a step stays fast. If the new tests live in a new folder, check that mypy and pytest still find their helpers (see the overrides in pyproject.toml).

Code standards

Ruff (rules ASYNC B C4 D E F I PIE PT RET RUF SIM UP W, Google docstrings, line length 100) and strict mypy over src, tests, tools, scripts and samples must pass.

  • Docstrings on every public module, class and function: what it does, why it exists, where it is used, in 2-4 lines. A multi-line docstring has a one-line summary, a blank line, then the body (D205). Args: / Returns: only when the types do not say it. Tests follow the same 2-4 lines even though lint does not check them.
  • from __future__ import annotations at the top of every module.
  • Names carry units: _secs (float seconds), _ms (int milliseconds), _bytes; counts are bare; booleans are positive (enabled, not disabled).
  • No side effects at import: no clients, network, file reads, threads or logging configuration. The app configures structlog; the library only logs. The one exception is os.register_at_fork in core.py (the fork guard): it must be in place before any fork, and it only marks the process's queues in a forked child.
  • Optional dependencies are imported lazily, at the point of use, and a missing one raises MissingExtra(feature, extra), which names the extra to install.
  • One error family. Everything raised to callers is an LLMQueueError. Bad config is ConfigError, raised at startup or on a refused registry edit. A failed call is CallFailed with a kind. Do not add new exception types where a kind will do.
  • Never raise into a live call because of bad config. Validate at boot or when the registry is published, list every problem at once, and fall back to the default at run time.
  • Performance claims come with numbers, measured, with the date, next to the claim (see the comments on hiredis, orjson, uuid-utils in pyproject.toml). If you have not measured it, write "unverified".
  • Keep the "why" comments that explain a trap, a measurement or a failure mode. Shorten them if they are long; do not delete them.
  • # noqa only for a selected rule, with the code and a reason on the same line.
  • mypy also runs with warn_unreachable and the ignore-without-code, redundant-expr and truthy-bool error codes. A # type: ignore names its code and gives a reason.
  • pre-commit (optional): .pre-commit-config.yaml runs ruff (check --fix and format, same version as uv.lock) and basic file hygiene. Run it with uvx pre-commit run --all-files, or install the git hook with uvx pre-commit install. scripts/check.py stays the full gate.

Library rules

  • No app or business config inside the library. Accounts, models, regions, budgets and keys come from the registry document (LLMQ_REGISTRY) and the LLMQ_* settings (LLMQueue.from_env, optionally a secrets= lookup). No defaults that only make sense for one consumer.
  • No compatibility shims. Removing or renaming a public name is a breaking change in CHANGELOG.md (STABILITY.md), not quietly mapped to the new one.
  • The public API is llm_queue.__all__, the registry setting names and the llm-queue commands. Changing any of them is a CHANGELOG entry.
  • Message format (v) and store key schema changes follow STABILITY.md: a worker returns a newer message to the queue instead of dropping it, and a schema change ships with a migration note.

Tests

Folder Runs Needs
tests/unit/ registry, clients, tokens and inference, fakellm nothing (default testpaths)
tests/integration/registry/ the registry store the stack
tests/integration/engine/ queue, admission, runtime, ops, streaming, front door the stack

Engine tests share engine_testkit.py (fakellm on sys.path, the test registry, stats) and these fixtures from tests/integration/engine/conftest.py:

  • store: an async client, parametrised over Valkey (6379) and Redis (6380), so every engine test runs on both engines. Do not narrow a test to one engine.
  • prefix: a unique key prefix per test, every key under it deleted at teardown.
  • make: builds an Engine with its own in-process fakellm (make(models, scenario)), and stops it at teardown.

Rules:

  • Pin a bug with a test that fails without the fix. Its docstring names the bug and what used to happen.
  • No fixed sleeps to wait for state. Poll for the condition with a deadline and fail with a clear message when the deadline passes. A sleep is fine only when time itself is the thing under test (a lease, a window).
  • Fixed-window (per-minute) budgets reset on the calendar minute. A test that counts against one must start away from the minute's end (see _away_from_minute_end in test_runtime.py) or check that the minute did not change, or it fails at random.
  • Test failure paths and cleanup (nothing left held, no keys left behind), not only the happy path.
  • pytest is strict (pyproject.toml): every warning is an error (filterwarnings = ["error", ...]), unknown markers and config keys fail (--strict-markers, --strict-config), and an xfail that passes fails (xfail_strict). Fix a new warning at its source; add a narrow ignore: filter, with a comment, only for a warning a third-party library raises. Register any new marker under markers.

Test tooling

  • fakellm knobs and counters. fakellm's behaviour is set per model x target key in a scenario (YAML or JSON at boot, PUT /admin/scenario) or patched mid-test (PATCH /admin/models/{key}); POST /admin/reset clears the counters and GET /admin/stats reads them. Besides latency, RPM / TPM, concurrency, throttle / 5xx / timeout rates and hang, a key can set:
Field Effect
force_error every call returns that ERRORS row; the rows include the auth errors (UnrecognizedClientException, InvalidSignatureException, InvalidSignatureClockSkew, ExpiredTokenException) and the quota error (ServiceQuotaExceededException, OpenAI insufficient_quota)
force_message replaces every error's message (to test classification by message)
cut_after_chunks, cut_with a stream ends after that many text chunks, with an in-stream exception, stall (no more bytes, socket open) or close (socket dropped)
tail_rate, tail_ms a heavy latency tail; latency_per_1k_input_ms makes latency grow with the prompt
reasoning reasoning blocks (text, redacted, both), gpt-oss style
cache_read_tokens, cache_write_tokens prompt-cache usage in the reply

Per key, /admin/stats also counts calls per wall-clock minute (by_minute, to check a fixed-window budget at the provider) and duplicates (a request body seen twice since the reset: a call that ran twice). - Seeds. scripts/scenarios.py, scripts/gates.py and scripts/soak.py seed every random choice and print the seed at the start; --seed N replays a run. - Perf baseline. Gate B34 compares the queue's p95 and ceiling with scripts/baselines/perf.json (within 25 % + 10 ms, ceiling at least 80 %) and fails when there is none. Record one per machine with python scripts/gates.py B34 --record-baseline. - Benchmark. python scripts/bench.py measures pod start, the cold first call, the first call after --idle seconds, steady and stream overhead and store round trips per call, with the public API only (--reps, --only PHASE..., --out FILE.json). The 0.1.0b1 numbers are in scripts/baselines/bench-0.1.0b1.json; compare only runs made back to back on one machine. - Property tests (Hypothesis). tests/unit/engine/test_budget_properties.py checks the token estimate, used tokens and refunds; tests/integration/engine/test_budget_clock.py checks the admission scripts' fixed and smooth RPM windows grant by grant against a Python model, at chosen times: Admission(clock_ms=...) (tests only) replaces the store's clock. - Environment. Scripts that start queues on the local stack set LLMQ_ENVIRONMENT=test (a plain-TCP store and no salt are refused otherwise, since unset is strict). - Timeouts and coverage. pytest-timeout fails a test after 120 s (timeout in pyproject.toml) instead of stalling CI; branch coverage must stay at or above 89 % (fail_under), measured by the Python 3.12 integration job.

Lua scripts

Admission (engine/_admission_lua.py) and other atomic store operations are Lua.

  • One script per atomic operation (PICK_TAKE, FINISH, UNDO, RENEW, SWEEP). Do not split an operation across round trips, and do not merge unrelated operations into one script.
  • Key names are built inside the script from the prefix in ARGV. This depends on the single-primary store the library supports (cluster mode is not supported); say so if you change it.
  • Time is the store's clock (redis.call('TIME')), never the caller's.
  • Measure a script change by its server time on the stack, before and after, and put the numbers in the change.

Performance changes

A change made for speed needs before and after numbers from the same machine: the gates.py gate it affects, load.py, or a micro-benchmark of the hot path (state the Python version and date). Record the result in the CHANGELOG Performance section. No numbers, no merge.

Docs site

The site (Zensical, GitHub Pages) is built from zensical.toml and follows Diátaxis: docs/tutorials/ (learning: getting started), docs/how-to/ (tasks: configure, tune, call, operate, share models, test), docs/explanation/ (how it works: architecture pages with Mermaid diagrams), docs/reference/ (facts: configuration, errors, CLI and HTTP, extension points, the API generated from the docstrings) and docs/project/ (changelog, stability, contributing, security, included from the root files). Put a new page in the section that matches what the reader is doing, and add it to nav. Internal docs (ARCHITECTURE.md and the git-ignored docs/internal/, which holds the design plan, walkthrough and release steps) are excluded from the site and from the sdist. README links are absolute GitHub URLs (it is shown on PyPI).

uvx --with "mkdocstrings-python==2.0.11" "zensical==0.1.1" build --clean --strict  # as CI
uvx --with "mkdocstrings-python==2.0.11" "zensical==0.1.1" serve                   # local preview

.github/workflows/docs.yml builds every pull request with --strict and deploys on a published release or a manual run (Actions > Docs > Run workflow).

Docs to update with a change

Doc When
README.md a public name, setting or behaviour changes
CHANGELOG.md always: under the unreleased version, in Added, Fixed, Performance, Removed or Changed
docs/internal/ACCOUNT_ROUTING_PLAN.md §13 (maintainers only) implementation status of a plan item changes
samples/README.md (and the sample) a usage pattern changes or a sample is added
docs/explanation/*.md the structure, the data flow, the failure model or a measured number changes

CHANGELOG entries are plain sentences: what changed and, for a fix, what used to go wrong.

Commits

  • Imperative subject under about 72 characters, a blank line, then a body that explains why.
  • No Co-Authored-By lines and no "Generated with" footers.
  • Stage only the files you changed. Never sweep in unrelated work.
  • Never commit .env, .env.* or .pypirc (they are git-ignored and kept out of the sdist by its allow-list; keep it that way).
  • Never skip hooks (--no-verify) or bypass signing.

Releases

Releases go out only by pushing a tag v<version>: .github/workflows/publish.yml checks that the tag matches __version__, is on main and has a CHANGELOG entry, re-runs CI, builds once, uploads to TestPyPI, smoke-tests that upload, uploads to PyPI with trusted publishing (OIDC, with provenance) and creates the GitHub Release. Never upload with twine or by hand, and never store a PyPI token. A manual run of the workflow is a dry run that stops after the TestPyPI smoke test. The steps are in docs/RELEASING.md (maintainers' notes in docs/internal, not published).