Skip to content

04 Sync

Call from sync code: scripts, Celery tasks, or the threads of a sync library. Async code should use converse / embed instead.

Key points

  • converse_sync / embed_sync run on the direct path in a private event-loop thread. They share the same RPM / TPM budgets and health as queued calls, but have no queue wait.
  • No start() is needed for sync calls.
  • Call close_sync() when done, to stop the private loop and close the clients.

Code

samples/04_sync.py
"""Call from sync code (scripts, Celery tasks, threads of a sync library).

`converse_sync` / `embed_sync` run on the direct path in a private event loop thread: same
RPM / TPM budgets and health as queued calls, but no queue wait. Use them only where an event
loop is not available; async code should use `converse` / `embed`.

    python scripts/samples.py 04_sync [--live]
"""

from __future__ import annotations

from llm_queue import LLMQueue


def main() -> None:
    llmq = LLMQueue.from_env()  # no start() needed for sync calls; nothing to await
    request = {
        "messages": [{"role": "user", "content": [{"text": "Reply with the word ready."}]}],
        "inferenceConfig": {"maxTokens": 200},
    }
    reply = llmq.converse_sync("gpt20b", request)
    blocks = reply.response["output"]["message"]["content"] if reply.response else []
    text = "".join(b["text"] for b in blocks if "text" in b)
    print(f"gpt20b (sync): {text.strip()[:80]!r} via {reply.meta.deployment}")
    vectors = llmq.embed_sync("titan", ["sync embedding"]).vectors
    print(f"titan (sync): {len(vectors)} vector of {len(vectors[0])} dims")
    assert text
    assert len(vectors) == 1
    llmq.close_sync()


if __name__ == "__main__":
    main()

Output (live, 2026-11-14)

gpt20b (sync): 'ready' via aws-main/us-east-1
titan (sync): 1 vector of 1024 dims