Hybrid Architecture in Practice

Mature RAG systems stop treating this as a choice. The two approaches compose cleanly, because they occupy different layers: blob storage holds truth, the vector index provides fast semantic lookup over that truth, and tools are the interface the model actually uses — with semantic search as simply one of the tools available to it.

The layered architecture:

1. Blob storage holds every original document plus its extracted text. Nothing else is a system of record.
2. The metadata catalog holds the queryable facts — titles, dates, sources, tenants, ACLs — and drives every filter.
3. The vector store indexes chunks of the extracted text, each carrying a doc_id, blob_key, and character offsets pointing back to layer 1. It is fully rebuildable, so it can be thrown away and regenerated at will.
4. The tool layer exposes semantic_search (backed by the vector index), search_documents (backed by full-text/BM25), list_documents and read_document (backed by blob storage plus the catalog). The model picks the right instrument for the question it was asked.

Retrieve then verify is the pattern that makes the combination worth more than either half. Semantic search is used for what it is genuinely good at — finding candidates — and its results are treated as pointers rather than as answers. When a chunk looks relevant, the model calls read_document on the surrounding span and reads the passage in its real context before citing it. That single move eliminates the chunk-context problem that most of the mitigations in this article exist to work around: the chunk finds the page, the document supplies the truth.

Routing keeps the cost sane. Cheap classification of the incoming question — does it name a specific document, contain an identifier, or ask a conceptual question? — decides whether to go straight to a direct lookup, run a lexical search, run a semantic search, or hand the model the full toolset for an open-ended investigation. Most production traffic is repetitive, and routing the easy 80% to a single fast path leaves the latency and token budget available for the questions that genuinely need multi-step retrieval.

What this buys you: exact lookups stay exact, conceptual questions still get semantic matching, permissions are enforced when a document is read rather than when it was indexed, the index can be rebuilt from scratch after any change to chunking or embedding models, and every answer traces back to a versioned object in blob storage that you can actually go and open.