08 LangChain¶
Drop-in LangChain chat model and embeddings whose calls go through the queue. Use them in LangChain / LangGraph chains, agents and RAG pipelines.
Key points¶
QueuedChatBedrockConverseis aChatBedrockConverse. Tools and structured output work as usual.- Streaming is off by default:
astream(),astream_events()and LangGraph'sstream_mode="messages"yield the whole answer as one chunk. Passqueue_streaming=True(onQueuedChatBedrockConverseorQueuedChatOpenAI) to get tokens as they arrive, throughLLMQueue.converse_stream/chat_stream(see Stream):- chunks are parsed by the installed langchain-aws / langchain-openai, so text deltas and
tool_call_chunks(index, id, name, argument pieces) look as from the parent class; usage_metadataarrives once, on the last chunk, with the stop reason (finish_reason; Bedrock'sstopReasontoo) and how the queue served the call (served_model,deployment,attempts, ...) inresponse_metadata;- a failure after tokens were sent raises
CallFailedwithpartial_text; - a consumer that stops reading cancels the queued call;
- sync
stream()still yields one whole message (the queue has no sync stream), and a sync call on the queue's own event loop raises the queue'sLLMQueueErrorunchanged: useawait chat.ainvoke(...)/astream(...)there; disable_streaming="tool_calling"is kept; langchain-aws 0.2 turns streaming off for a Bedrock model id it does not know (passdisable_streaming=Falseto stream anyway);- a
QueuedChatOpenAIstructured-output request (response_format) is answered whole.
- chunks are parsed by the installed langchain-aws / langchain-openai, so text deltas and
QueuedBedrockEmbeddingsis aBedrockEmbeddings.- Both find the started
LLMQueueon their own.model_aliasnames a registry model, and themodel_idcomes from the registry. - They work with LangChain 1.x; CI also checks the 0.3 line.
Code¶
samples/08_langchain.py
"""LangChain: drop-in chat model and embeddings that go through the queue (chains, agents, RAG).
`QueuedChatBedrockConverse` is a `ChatBedrockConverse` whose calls go through the queue (tools
and structured output work as usual; `astream()` yields the whole answer as one chunk unless
`queue_streaming=True`, which streams tokens through `LLMQueue.converse_stream`);
`QueuedBedrockEmbeddings` is a `BedrockEmbeddings`. They find the started `LLMQueue` on their own.
python scripts/samples.py 08_langchain [--live]
"""
from __future__ import annotations
import asyncio
from langchain_core.messages import AIMessage, AIMessageChunk
from langchain_core.tools import tool
from llm_queue import LLMQueue
from llm_queue.langchain import QueuedBedrockEmbeddings, QueuedChatBedrockConverse
@tool
def stock_price(symbol: str) -> str:
"""Latest price of an NSE stock symbol."""
return f"{symbol}: 2,950.10"
async def main() -> None:
llmq = LLMQueue.from_env()
await llmq.start()
try:
# model_id comes from the registry of the started queue
chat = QueuedChatBedrockConverse(model_alias="gpt120b", max_tokens=400) # type: ignore[call-arg]
message = await chat.ainvoke("In one line, what is an index fund?")
print(f"chat: {message.text()[:120]!r} via {message.response_metadata['deployment']}")
with_tools = chat.bind_tools([stock_price])
called = await with_tools.ainvoke("Use the stock_price tool to get the price of RELIANCE.")
assert isinstance(called, AIMessage)
print(f"tool calls: {[(c['name'], c['args']) for c in called.tool_calls]}")
assert called.tool_calls
assert called.tool_calls[0]["name"] == "stock_price"
# queue_streaming=True: astream() yields tokens as they arrive (through converse_stream)
streaming = QueuedChatBedrockConverse( # type: ignore[call-arg]
model_alias="gpt120b", max_tokens=400, queue_streaming=True
)
chunks: list[AIMessageChunk] = []
async for chunk in streaming.astream("Name three Nifty 50 companies, comma separated."):
chunks.append(chunk)
streamed = sum(chunks[1:], chunks[0])
usage = streamed.usage_metadata # once, on the last chunk
assert usage is not None
print(
f"streamed: {len(chunks)} chunks, {streamed.text()[:80]!r}, "
f"finish_reason={streamed.response_metadata.get('finish_reason')}, "
f"output_tokens={usage['output_tokens']}"
)
emb = QueuedBedrockEmbeddings(model_alias="titan")
docs = await emb.aembed_documents(["first", "second"])
query = await emb.aembed_query("which one")
print(f"embeddings: {len(docs)} docs, query of {len(query)} dims")
assert len(docs) == 2
assert query
finally:
await llmq.stop()
if __name__ == "__main__":
asyncio.run(main())