Skip to content

08 LangChain

Drop-in LangChain chat model and embeddings whose calls go through the queue. Use them in LangChain / LangGraph chains, agents and RAG pipelines.

Key points

  • QueuedChatBedrockConverse is a ChatBedrockConverse. Tools and structured output work as usual.
  • Streaming is off by default: astream(), astream_events() and LangGraph's stream_mode="messages" yield the whole answer as one chunk. Pass queue_streaming=True (on QueuedChatBedrockConverse or QueuedChatOpenAI) to get tokens as they arrive, through LLMQueue.converse_stream / chat_stream (see Stream):
    • chunks are parsed by the installed langchain-aws / langchain-openai, so text deltas and tool_call_chunks (index, id, name, argument pieces) look as from the parent class;
    • usage_metadata arrives once, on the last chunk, with the stop reason (finish_reason; Bedrock's stopReason too) and how the queue served the call (served_model, deployment, attempts, ...) in response_metadata;
    • a failure after tokens were sent raises CallFailed with partial_text;
    • a consumer that stops reading cancels the queued call;
    • sync stream() still yields one whole message (the queue has no sync stream), and a sync call on the queue's own event loop raises the queue's LLMQueueError unchanged: use await chat.ainvoke(...) / astream(...) there;
    • disable_streaming="tool_calling" is kept; langchain-aws 0.2 turns streaming off for a Bedrock model id it does not know (pass disable_streaming=False to stream anyway);
    • a QueuedChatOpenAI structured-output request (response_format) is answered whole.
  • QueuedBedrockEmbeddings is a BedrockEmbeddings.
  • Both find the started LLMQueue on their own. model_alias names a registry model, and the model_id comes from the registry.
  • They work with LangChain 1.x; CI also checks the 0.3 line.

Code

samples/08_langchain.py
"""LangChain: drop-in chat model and embeddings that go through the queue (chains, agents, RAG).

`QueuedChatBedrockConverse` is a `ChatBedrockConverse` whose calls go through the queue (tools
and structured output work as usual; `astream()` yields the whole answer as one chunk unless
`queue_streaming=True`, which streams tokens through `LLMQueue.converse_stream`);
`QueuedBedrockEmbeddings` is a `BedrockEmbeddings`. They find the started `LLMQueue` on their own.

    python scripts/samples.py 08_langchain [--live]
"""

from __future__ import annotations

import asyncio

from langchain_core.messages import AIMessage, AIMessageChunk
from langchain_core.tools import tool

from llm_queue import LLMQueue
from llm_queue.langchain import QueuedBedrockEmbeddings, QueuedChatBedrockConverse


@tool
def stock_price(symbol: str) -> str:
    """Latest price of an NSE stock symbol."""
    return f"{symbol}: 2,950.10"


async def main() -> None:
    llmq = LLMQueue.from_env()
    await llmq.start()
    try:
        # model_id comes from the registry of the started queue
        chat = QueuedChatBedrockConverse(model_alias="gpt120b", max_tokens=400)  # type: ignore[call-arg]
        message = await chat.ainvoke("In one line, what is an index fund?")
        print(f"chat: {message.text()[:120]!r} via {message.response_metadata['deployment']}")

        with_tools = chat.bind_tools([stock_price])
        called = await with_tools.ainvoke("Use the stock_price tool to get the price of RELIANCE.")
        assert isinstance(called, AIMessage)
        print(f"tool calls: {[(c['name'], c['args']) for c in called.tool_calls]}")
        assert called.tool_calls
        assert called.tool_calls[0]["name"] == "stock_price"

        # queue_streaming=True: astream() yields tokens as they arrive (through converse_stream)
        streaming = QueuedChatBedrockConverse(  # type: ignore[call-arg]
            model_alias="gpt120b", max_tokens=400, queue_streaming=True
        )
        chunks: list[AIMessageChunk] = []
        async for chunk in streaming.astream("Name three Nifty 50 companies, comma separated."):
            chunks.append(chunk)
        streamed = sum(chunks[1:], chunks[0])
        usage = streamed.usage_metadata  # once, on the last chunk
        assert usage is not None
        print(
            f"streamed: {len(chunks)} chunks, {streamed.text()[:80]!r}, "
            f"finish_reason={streamed.response_metadata.get('finish_reason')}, "
            f"output_tokens={usage['output_tokens']}"
        )

        emb = QueuedBedrockEmbeddings(model_alias="titan")
        docs = await emb.aembed_documents(["first", "second"])
        query = await emb.aembed_query("which one")
        print(f"embeddings: {len(docs)} docs, query of {len(query)} dims")
        assert len(docs) == 2
        assert query
    finally:
        await llmq.stop()


if __name__ == "__main__":
    asyncio.run(main())

Output (live, 2026-11-14)

chat: 'An index fund is a passively managed mutual fund or ETF that seeks to replicate the performance of a specific market ind' via aws-main/us-east-1
tool calls: [('stock_price', {'symbol': 'RELIANCE'})]
embeddings: 2 docs, query of 1024 dims