Semantic Search with Embeddings: From Zero to Production

Semantic search matches on meaning instead of keywords. "How do I reset my password?" finds the doc titled "Account recovery" even though they share no words. The core idea is simple: convert text to vectors where similar meanings are close together, then find the nearest neighbors. The engineering is in the details: which embedding model, which index, and how you measure whether retrieval is actually good.

How Embeddings Work

An embedding model maps text to a fixed-size vector (e.g. 1536 dimensions for text-embedding-3-small). The model is trained so that semantically similar texts produce vectors that are close under cosine similarity:

"how do I reset my password"  ──►  [0.12, -0.45, 0.87, ...]
"account recovery steps"      ──►  [0.14, -0.42, 0.85, ...]   ← close
"pizza delivery near me"      ──►  [-0.31, 0.72, -0.19, ...]  ← far

Choosing an Embedding Model

Model	Dimensions	Cost	Best For
text-embedding-3-small	1536	Low	General purpose, cost-sensitive
text-embedding-3-large	3072	Higher	Maximum quality
bge-m3 / multilingual	1024	Self-hosted	Multilingual, on-prem
voyage-3 / cohere embed	1024	API	Long documents, code

Rules:

• Test on your data, not benchmarks. Embedding quality is domain-specific.
• Match the model at index and query time: mixing models produces garbage similarity.
• Consider multilingual needs up front; switching models means re-indexing everything.
• Code search benefits from code-specific models; general text models miss code semantics.

Indexing with pgvector

If you already run PostgreSQL, pgvector adds vector columns without a new infrastructure component:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE documents (
    id BIGSERIAL PRIMARY KEY,
    title TEXT NOT NULL,
    body TEXT NOT NULL,
    embedding vector(1536)
);

CREATE INDEX ON documents USING hnsw (embedding vector_cosine_ops);

Indexing from Python:

import psycopg
from openai import OpenAI

client = OpenAI()

def index_document(title: str, body: str):
    text = f"{title}\n{body}"
    embedding = client.embeddings.create(
        model="text-embedding-3-small", input=text
    ).data[0].embedding

    with psycopg.connect("dbname=search") as conn:
        conn.execute(
            "INSERT INTO documents (title, body, embedding) VALUES (%s, %s, %s)",
            (title, body, embedding),
        )

Querying:

def search(query: str, top_k: int = 5) -> list[dict]:
    q_embedding = client.embeddings.create(
        model="text-embedding-3-small", input=query
    ).data[0].embedding

    with psycopg.connect("dbname=search") as conn:
        rows = conn.execute(
            """
            SELECT title, body, 1 - (embedding <=> %s::vector) AS similarity
            FROM documents
            ORDER BY embedding <=> %s::vector
            LIMIT %s
            """,
            (q_embedding, q_embedding, top_k),
        ).fetchall()
    return [{"title": r[0], "body": r[1], "score": r[2]} for r in rows]

<=> is cosine distance in pgvector; 1 - distance converts it to a similarity score.

Indexing with FAISS

For large corpora (millions of vectors) or when you want to avoid database coupling, FAISS is the standard library:

import faiss
import numpy as np

DIM = 1536
index = faiss.IndexFlatIP(DIM)  # inner product = cosine for normalized vectors

def normalize(v: list[float]) -> np.ndarray:
    arr = np.array(v, dtype="float32")
    return arr / np.linalg.norm(arr)

def add_documents(embeddings: list[list[float]], ids: list[int]):
    vectors = np.vstack([normalize(e) for e in embeddings])
    index.add_with_ids(vectors, np.array(ids, dtype="int64"))

def search(query_embedding: list[float], top_k: int = 5):
    q = normalize(query_embedding).reshape(1, -1)
    scores, ids = index.search(q, top_k)
    return list(zip(ids[0].tolist(), scores[0].tolist()))

For millions of vectors, replace IndexFlatIP (exact, brute force) with an HNSW index:

index = faiss.IndexHNSWFlat(DIM, 32)  # 32 neighbors per node
index.hnsw.efConstruction = 200       # build quality
index.hnsw.efSearch = 64              # query recall vs speed

Chunking Matters More Than the Model

Semantic search operates on chunks, not whole documents. A 50-page manual embedded as one vector is useless: the average of everything is nothing. Chunk by logical units (sections, paragraphs) with overlap:

def chunk_document(text: str, chunk_size: int = 800, overlap: int = 100) -> list[str]:
    chunks = []
    start = 0
    while start < len(text):
        end = min(start + chunk_size, len(text))
        chunks.append(text[start:end])
        start = end - overlap
    return chunks

Store the chunk's source document and section in metadata so results can cite their origin.

Evaluating Retrieval Quality

Retrieval quality is measurable. Build a small labeled set of queries with the documents that should be returned, then compute:

• Recall@k: fraction of relevant documents found in the top-k.
• MRR (Mean Reciprocal Rank): how high the first relevant result ranks.
• Hit rate: did any relevant document appear in the top-k?

def recall_at_k(results: list[str], relevant: set[str], k: int) -> float:
    top_k = set(results[:k])
    return len(top_k & relevant) / len(relevant)

def mrr(results: list[str], relevant: set[str]) -> float:
    for i, doc in enumerate(results, start=1):
        if doc in relevant:
            return 1.0 / i
    return 0.0

Run this on every change: new embedding model, new chunk size, new index parameters. A 2% recall drop from a "better" model is a regression, not an upgrade.

Production Considerations

1. Re-index strategy: embeddings change when you upgrade models. Version your embeddings and rebuild indexes in the background, then swap atomically.
2. Metadata filtering: combine vector search with filters (date, tenant, doc type) to keep results scoped.
3. Hybrid search: keyword + vector fusion usually beats either alone (see the hybrid search guide).
4. Caching: cache embeddings for repeated queries; embedding calls are cheap but add latency.
5. Monitoring: track p95 query latency, index size, and recall on a shadow set.

Implementation Checklist

• Test 2-3 embedding models on your own data before choosing
• Use the same model for indexing and querying
• Chunk documents by logical units with overlap
• Store source metadata with every chunk
• Use HNSW for large corpora, flat index for small ones
• Build a labeled query set and measure recall@k and MRR
• Version embeddings and rebuild indexes in the background
• Add metadata filters to scope results
• Consider hybrid keyword + vector search

---

MatterAI builds frontier AI infrastructure for engineering teams — from inference-optimized models to autonomous coding agents and agentic code reviews.

Explore what we're building:

• Orbital IDE — Autonomous AI coding agent with background agents and deep codebase memory
• AI Code Reviews — Agentic pre-commit reviews across GitHub, GitLab, and Bitbucket
• Axon Models — Frontier-grade reasoning models at 70% lower inference cost

Get started free - https://app.matterai.so

• Read the docs
• View pricing

---

Follow us on X · LinkedIn · GitHub