Skip to content

03 Embed

Embed texts with Titan and Cohere. Use it to index documents and to embed search queries.

Key points

  • embed returns one vector per text, in order.
  • The library splits the texts into batches of the model's embed_batch_size: Titan takes 1 text per request, Cohere up to 96. Every batch goes through the queue and the model's RPM budget.
  • Cohere needs input_type: search_document when indexing, search_query when searching.

Code

samples/03_embed.py
"""Embed texts with Titan and Cohere (search indexing, retrieval, follow-up cards).

`embed` returns one vector per text, in order. The library splits the texts into batches of
the model's `embed_batch_size` (Titan takes 1 text per request, Cohere up to 96), and every
batch goes through the queue and the model's RPM budget. Cohere needs `input_type`:
`search_document` when indexing, `search_query` when searching.

    python scripts/samples.py 03_embed [--live]
"""

from __future__ import annotations

import asyncio
import math

from llm_queue import LLMQueue

TEXTS = [
    "Nifty 50 closed higher on banking gains.",
    "Banking stocks lifted the Nifty index today.",
    "The monsoon arrived early in Kerala.",
]


def cosine(a: list[float], b: list[float]) -> float:
    dot = sum(x * y for x, y in zip(a, b, strict=True))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))


async def main() -> None:
    llmq = LLMQueue.from_env()
    await llmq.start()
    try:
        for model, input_type in (("titan", None), ("cohere", "search_document")):
            result = await llmq.embed(model, TEXTS, input_type=input_type)
            vectors = result.vectors
            assert len(vectors) == len(TEXTS), f"{model}: {len(vectors)} vectors"
            close, far = cosine(vectors[0], vectors[1]), cosine(vectors[0], vectors[2])
            print(
                f"{model}: {len(vectors)} vectors of {len(vectors[0])} dims; "
                f"similar pair {close:.3f}, unrelated pair {far:.3f}"
            )
        query = await llmq.embed("cohere", ["bank index rally"], input_type="search_query")
        print(f"cohere query vector: {len(query.vectors[0])} dims")
    finally:
        await llmq.stop()


if __name__ == "__main__":
    asyncio.run(main())

Output (live, 2026-11-14)

titan: 3 vectors of 1024 dims; similar pair 0.727, unrelated pair 0.069
cohere: 3 vectors of 1024 dims; similar pair 0.829, unrelated pair 0.365
cohere query vector: 1024 dims

The two banking sentences score close on both models, and the monsoon sentence scores far. Titan separates them more sharply.