Vector Stores: Embedding-Based Retrieval

The vector store approach front-loads the work. Documents are chunked and embedded during ingestion, and at query time the user's question is embedded with the same model and matched against the index by vector similarity. The top-k results are injected into the prompt before the model ever sees the question. One embedding call plus one index lookup, and retrieval is done.

What a record actually contains — four parts, and only the first is a vector:

• The vector: a fixed-length float array (384 to 3072 dimensions, depending on the embedding model) that positions this chunk in semantic space.
• The chunk text: the passage itself, stored alongside so retrieval doesn't require a second round trip to fetch what it just found.
• The metadata payload: doc_id, source, date, author, tenant, tags, ACL labels — everything you need to filter on.
• The pointer home: the blob key and character offsets, so the model (or a follow-up tool call) can always go read the full document the chunk came from.

Choosing a store falls into two camps. Dedicated vector databases — Pinecone, Qdrant, Weaviate, Milvus, Chroma, Vespa — offer the best index performance, hybrid search, and scaling story. Extensions to a database you already run — pgvector for Postgres, Elasticsearch/OpenSearch, Redis, MongoDB Atlas, sqlite-vec or FAISS for local work — avoid a new piece of infrastructure and keep vectors transactionally consistent with the rest of your data. For most teams under a few million chunks, pgvector in the Postgres you already operate is the right answer, and moving later is straightforward because you can rebuild the index from blob storage.

Index structures determine the speed/accuracy/memory triangle: flat (exact brute force, perfect recall, fine up to ~100k vectors), HNSW (graph-based, fast and high-recall, but the whole graph lives in RAM), and IVF-PQ (clustered plus compressed, enabling billion-scale search on modest hardware at some recall cost). These are covered in depth under Retrieval Mechanics later in this article.

Metadata filtering is where production systems live or die. Filtering after the similarity search (post-filter) can return nothing when the top-k all fail the filter; filtering before or during the search (pre-filter) is what you want, and support varies sharply between stores. For multi-tenancy, use separate collections or namespaces per tenant rather than relying on a metadata field — a filter bug leaks another customer's documents, a namespace boundary does not.

Operational realities that surprise people:

• The re-embedding problem: change your embedding model and every vector in the store is worthless. Old and new vectors are not comparable. Plan for full rebuilds — write into a new collection, verify, then swap — which is only painless if blob storage still holds every original.
• Idempotent upserts: derive chunk IDs deterministically (sha256(doc_id + chunk_index)) so re-ingesting a document overwrites its chunks instead of duplicating them. Duplicated chunks silently degrade results by filling top-k with the same passage.
• Deletes are a real workflow: when a document is removed or superseded, its chunks must be deleted by doc_id filter. Systems that forget this happily cite documents that no longer exist.
• Freshness lag: there is always a window between a document changing and the index reflecting it — minutes for a good pipeline, days for a nightly batch. Users experience this as the system "lying" about recent changes.
• Memory is the cost driver: an HNSW index of 10 million 1536-dimension vectors is roughly 60 GB of RAM before overhead. Quantization (scalar or binary) cuts this dramatically for a few points of recall.
• Recall is tunable, not fixed: ef_search (HNSW) and nprobe (IVF) trade latency for accuracy at query time. Ship with a measured evaluation set, not a default value someone copied from a blog post.

Advantages: sub-100ms retrieval regardless of corpus size, one predictable model call per question, genuine semantic matching (finds "terminated the agreement" when the user asked about "cancelling a contract"), scales to hundreds of millions of chunks, and deterministic enough to evaluate and cache.

Disadvantages: an ingestion pipeline to build and operate, embedding costs on every document and every re-index, staleness, weakness on exact identifiers and numbers, no ability to follow a reference or iterate, chunks that arrive stripped of surrounding context, and permissions that must be encoded into the index at build time.

Implementation Example: