Retrieval Augmented Generation (RAG), Storage & Retrieval Architecture

Storage & Retrieval Architecture

Chunking strategies and embedding models are only half of a RAG system. Something has to physically hold the documents, and something has to hand the right pieces of them to the model at question time. That storage-and-retrieval layer is the part that decides whether your system is accurate, fresh, affordable, and auditable — and there are two fundamentally different ways to build it.

Approach 1 — Document store + tools (agentic retrieval): keep documents whole in blob storage, expose a small set of tools (list, search, read) to the model, and let it navigate the corpus the way a person would. Retrieval becomes an action the model takes, decided at inference time.

Approach 2 — Vector store (embedding retrieval): chunk everything ahead of time, embed each chunk, and store the vectors in an index. A similarity search runs before the model does, and the top matches are pasted into the prompt. Retrieval becomes a preprocessing step.

Neither is a strict upgrade over the other. They fail in different places, cost money in different places, and are correct for different kinds of questions. The rest of this section covers how each one is actually built, and how their characteristics compare.

Blob Storage: The Document Layer

Blob (object) storage — Amazon S3, Google Cloud Storage, Azure Blob Storage, Cloudflare R2, or self-hosted MinIO — is where the raw bytes of your corpus belong: the original PDFs, HTML captures, transcripts, spreadsheets, and images. It is cheap per gigabyte, effectively unlimited, extremely durable, and versioned, which makes it the natural source of truth that every other component points back to.

The critical property to understand is what blob storage cannot do. It is a key-value store: you can GET a key and you can LIST a prefix. You cannot ask it a question about content. That means your key layout is your index — the folder structure, naming convention, and accompanying metadata are the entire navigational surface that a retrieval tool (or an LLM) will have to work with.

Key layout: design paths so that a prefix listing is already a meaningful filter. A layout like tenant/source/domain/year/doc-id/version/file.pdf lets you scope a search to one customer, one data source, or one year with a single LIST call, and it makes per-prefix IAM policies possible.

Metadata catalog: pair the bucket with a row-per-document table in a database you can actually query (Postgres, DynamoDB, SQLite for small corpora). Store the object key, title, source URL, author, created/updated timestamps, MIME type, size, content checksum, tags, access-control labels, and pointers to derived artifacts. Filtering, sorting, and permission checks happen here — never by scanning the bucket.

Derived artifacts: ingestion is a pipeline, and each stage should be persisted next to the original: raw/report.pdf → extracted/report.md → chunks/report.jsonl. Parsing a PDF is slow and expensive; do it once, store the result, and let every downstream consumer read the clean text.

Access and lifecycle: serve documents through short-lived presigned URLs rather than making buckets public, scope IAM credentials per prefix, enable object versioning so a re-ingest never destroys history, and use lifecycle rules to move cold originals to archival tiers while keeping the extracted text hot.

What belongs where — a division of responsibility that keeps systems recoverable:

Blob storage: the bytes. The only place the original document exists. Immutable and versioned.
Metadata catalog: the facts you filter, sort, and authorize on. Cheap to query, easy to reindex.
Vector store: embeddings plus a pointer back to the blob key and character offsets — an index, not a database of record.

The rule that saves you: never let the vector store be the only copy of anything. If you can rebuild the entire index from blob storage plus the catalog with one job, then changing your chunking strategy or your embedding model is a Tuesday afternoon. If you can't, it's a migration project.

Storing images and multimodal content: blobs handle this naturally — the image itself stays in the bucket, while the catalog holds a generated caption or description, and the vector store (if used) embeds that description. The model receives the caption during search and can fetch the actual image by key when it needs to look at it.

Implementation Example:

Agentic Retrieval: Giving the Model Tools

In the tool-calling approach, nothing is retrieved before the model runs. Instead the model is handed a small set of functions over your document store and decides for itself what to look at — searching, reading, noticing a reference, and following it, exactly the way a research assistant would work through a filing cabinet.

The minimal tool set covers four verbs, and almost every agentic RAG system is some variation of them:

list_documents(prefix, filters, limit) — browse the corpus structure. Returns titles, IDs, dates, and sizes so the model can orient itself before reading anything.
search_documents(query, filters, limit) — find candidates by keyword, full-text (BM25), or metadata match. Returns snippets with document IDs, not whole files.
read_document(doc_id, offset, limit) — read actual content, paginated. This is the tool people get wrong: without offset/limit, a single 300-page PDF blows the context window and ends the conversation.
get_metadata(doc_id) — authorship, dates, source, version, related documents. Cheap, and often enough to rule a document in or out without reading it.

Structure is what makes this work. The model has no embeddings to lean on, so the organisation of the corpus is the retrieval quality. A manifest or table-of-contents file it reads first, descriptive file names, a short README per folder, consistent front matter (title, date, summary, tags) at the top of each document, stable IDs, and section anchors so reads can be targeted rather than sequential — these do the work that an embedding index would otherwise do.

The loop: the model emits a tool call, your code executes it against blob storage and the catalog, the result comes back as a tool result, and the model either calls another tool or answers. Two to five iterations is typical; a good system prompt telling the model what the corpus contains and how it is organised cuts that number substantially.

Advantages:

No ingestion pipeline: no chunking, no embedding costs, no index to rebuild. Point the tools at the store and it works.
Always fresh: the model reads the file as it exists right now. There is no staleness window between a document changing and answers changing.
Exact lookups are exact: "the Q3 invoice from Acme Corp" or "error code E-4471" resolves by name or by literal match — precisely the queries where vector similarity is weakest.
Multi-hop reasoning: the model can read a document, notice it cites another, and go fetch that one. A single vector search cannot follow a reference.
Permissions at read time: authorization is checked when the tool executes, using the caller's identity — not baked into an index that was built with someone else's access.
Whole-document context: the model reads a section in place, with its surrounding headings and caveats intact, instead of an isolated chunk stripped of its context.
Auditable: the transcript shows exactly which files were opened and in what order. Debugging a bad answer means reading the tool calls.

Disadvantages:

Latency and token cost: each tool call is a full round trip to the model. Three iterations of search-then-read can mean several seconds and tens of thousands of input tokens.
Depends on model reasoning: a weaker model searches badly, gives up early, or reads the wrong file. The retrieval quality is only as good as the planner.
Needs a good search tool or a bounded corpus: browsing works for hundreds or thousands of well-organised documents. Across ten million files with no search index, the model is lost.
Non-deterministic: the same question can take different paths on different runs, which complicates evaluation and caching.

Best Use Cases: codebases and repositories, structured document libraries with clear naming, legal and financial records where the exact document matters more than a semantically similar one, corpora that change constantly, and any system where per-user permissions must be enforced at query time.

Implementation Example:

MCP (Model Context Protocol) is the emerging standard for exposing exactly this kind of tool surface. Rather than hand-wiring tools into one application, you run a server that publishes list/search/read over your document store, and any MCP-compatible client can use it. See the dedicated section at the end of this article.

Vector Stores: Embedding-Based Retrieval

The vector store approach front-loads the work. Documents are chunked and embedded during ingestion, and at query time the user's question is embedded with the same model and matched against the index by vector similarity. The top-k results are injected into the prompt before the model ever sees the question. One embedding call plus one index lookup, and retrieval is done.

What a record actually contains — four parts, and only the first is a vector:

The vector: a fixed-length float array (384 to 3072 dimensions, depending on the embedding model) that positions this chunk in semantic space.
The chunk text: the passage itself, stored alongside so retrieval doesn't require a second round trip to fetch what it just found.
The metadata payload: doc_id, source, date, author, tenant, tags, ACL labels — everything you need to filter on.
The pointer home: the blob key and character offsets, so the model (or a follow-up tool call) can always go read the full document the chunk came from.

Choosing a store falls into two camps. Dedicated vector databases — Pinecone, Qdrant, Weaviate, Milvus, Chroma, Vespa — offer the best index performance, hybrid search, and scaling story. Extensions to a database you already run — pgvector for Postgres, Elasticsearch/OpenSearch, Redis, MongoDB Atlas, sqlite-vec or FAISS for local work — avoid a new piece of infrastructure and keep vectors transactionally consistent with the rest of your data. For most teams under a few million chunks, pgvector in the Postgres you already operate is the right answer, and moving later is straightforward because you can rebuild the index from blob storage.

Index structures determine the speed/accuracy/memory triangle: flat (exact brute force, perfect recall, fine up to ~100k vectors), HNSW (graph-based, fast and high-recall, but the whole graph lives in RAM), and IVF-PQ (clustered plus compressed, enabling billion-scale search on modest hardware at some recall cost). These are covered in depth under Retrieval Mechanics later in this article.

Metadata filtering is where production systems live or die. Filtering after the similarity search (post-filter) can return nothing when the top-k all fail the filter; filtering before or during the search (pre-filter) is what you want, and support varies sharply between stores. For multi-tenancy, use separate collections or namespaces per tenant rather than relying on a metadata field — a filter bug leaks another customer's documents, a namespace boundary does not.

Operational realities that surprise people:

The re-embedding problem: change your embedding model and every vector in the store is worthless. Old and new vectors are not comparable. Plan for full rebuilds — write into a new collection, verify, then swap — which is only painless if blob storage still holds every original.
Idempotent upserts: derive chunk IDs deterministically (sha256(doc_id + chunk_index)) so re-ingesting a document overwrites its chunks instead of duplicating them. Duplicated chunks silently degrade results by filling top-k with the same passage.
Deletes are a real workflow: when a document is removed or superseded, its chunks must be deleted by doc_id filter. Systems that forget this happily cite documents that no longer exist.
Freshness lag: there is always a window between a document changing and the index reflecting it — minutes for a good pipeline, days for a nightly batch. Users experience this as the system "lying" about recent changes.
Memory is the cost driver: an HNSW index of 10 million 1536-dimension vectors is roughly 60 GB of RAM before overhead. Quantization (scalar or binary) cuts this dramatically for a few points of recall.
Recall is tunable, not fixed: ef_search (HNSW) and nprobe (IVF) trade latency for accuracy at query time. Ship with a measured evaluation set, not a default value someone copied from a blog post.

Advantages: sub-100ms retrieval regardless of corpus size, one predictable model call per question, genuine semantic matching (finds "terminated the agreement" when the user asked about "cancelling a contract"), scales to hundreds of millions of chunks, and deterministic enough to evaluate and cache.

Disadvantages: an ingestion pipeline to build and operate, embedding costs on every document and every re-index, staleness, weakness on exact identifiers and numbers, no ability to follow a reference or iterate, chunks that arrive stripped of surrounding context, and permissions that must be encoded into the index at build time.

Implementation Example:

Characteristics & Trade-offs

The two approaches are not competing implementations of the same idea — they have genuinely different shapes, and comparing them dimension by dimension is the fastest way to know which one your problem needs.

Query style

Tool calling: iterative and exploratory. The model can search, look at what came back, and refine — or read one document and then go find the one it cites.
Vector store: single-shot. One query vector produces one ranked list. Follow-up requires a second application-level round.

Latency

Tool calling: 2–10 seconds typical, because each iteration is a full model round trip. Latency grows with question difficulty.
Vector store: 50–200 ms for retrieval, effectively constant no matter how large the corpus grows.

Cost model

Tool calling: no ingestion cost at all; cost is paid per query in input tokens, and long documents are expensive to read.
Vector store: heavy upfront cost (embed every chunk) plus ongoing index hosting/RAM; per-query cost is small and stable.

Freshness

Tool calling: perfect. The tool reads the current file.
Vector store: lagging by however long your ingestion pipeline takes — and silently wrong in the gap.

Exact matches (invoice numbers, error codes, product SKUs, names, dates)

Tool calling: excellent. Literal search and direct addressing are what it does.
Vector store: notoriously weak. Embeddings encode meaning, and identifiers have no meaning to encode — this is the main reason hybrid BM25 + vector search exists.

Conceptual/fuzzy questions ("what are our obligations if a customer churns early?")

Tool calling: dependent on the model guessing the right search terms and file names.
Vector store: excellent, and the entire reason embeddings were adopted — it matches paraphrases and synonyms the user never typed.

Scale

Tool calling: comfortable to roughly thousands of well-organised documents; needs a strong search backend beyond that.
Vector store: hundreds of millions of chunks with mature, well-understood operational patterns.

Setup and maintenance

Tool calling: hours. Write four tools over storage you already have.
Vector store: weeks. Chunking strategy, embedding pipeline, index tuning, re-embedding runbook, evaluation set.

Permissions

Tool calling: enforced at read time with the caller's identity — the natural, safe model.
Vector store: encoded into the index at build time. Workable with namespaces and pre-filters, but every permission change is an index concern.

Context quality

Tool calling: the model sees passages in situ, with headings, caveats, and neighbouring sections intact.
Vector store: isolated chunks, which is precisely why contextual retrieval and hierarchical chunking (covered elsewhere in this article) exist as mitigations.

Explainability

Tool calling: the transcript is the audit trail — you can see every file it opened.
Vector store: you get similarity scores, which tell you that something ranked highly but not why it was the right document.

Characteristic failure mode

Tool calling: the model searches badly, gives up, and answers from its own parametric knowledge — confidently and without citations.
Vector store: the right chunk exists but ranks 21st with k=20, and the model answers from irrelevant context it was handed.

Choosing, in practice:

Start with tools over blob storage when your corpus is under a few thousand documents, is well-named and well-structured, changes frequently, requires per-user permissions, or when exact-document lookup is the dominant query. It is dramatically cheaper to build and you will learn what your users actually ask before committing to a chunking strategy.
Reach for a vector store when the corpus is large, queries are conceptual rather than nominal, latency budgets are tight (sub-second, user-facing chat), volume makes per-query token cost prohibitive, or content is unstructured enough that names and folders carry no information.
Watch for the crossover signals: agentic retrieval that regularly needs more than four or five tool calls per question is telling you it needs a semantic index; a vector system where users keep asking for documents by name is telling you it needs tools.
A useful sequencing heuristic: build the tool-based version first because it is a day's work, measure which questions it fails, and add the vector index for exactly those. The reverse order tends to produce a large embedding pipeline serving queries that a filename lookup would have answered.

Hybrid Architecture in Practice

Mature RAG systems stop treating this as a choice. The two approaches compose cleanly, because they occupy different layers: blob storage holds truth, the vector index provides fast semantic lookup over that truth, and tools are the interface the model actually uses — with semantic search as simply one of the tools available to it.

The layered architecture:

Blob storage holds every original document plus its extracted text. Nothing else is a system of record.
The metadata catalog holds the queryable facts — titles, dates, sources, tenants, ACLs — and drives every filter.
The vector store indexes chunks of the extracted text, each carrying a doc_id, blob_key, and character offsets pointing back to layer 1. It is fully rebuildable, so it can be thrown away and regenerated at will.
The tool layer exposes semantic_search (backed by the vector index), search_documents (backed by full-text/BM25), list_documents and read_document (backed by blob storage plus the catalog). The model picks the right instrument for the question it was asked.

Retrieve then verify is the pattern that makes the combination worth more than either half. Semantic search is used for what it is genuinely good at — finding candidates — and its results are treated as pointers rather than as answers. When a chunk looks relevant, the model calls read_document on the surrounding span and reads the passage in its real context before citing it. That single move eliminates the chunk-context problem that most of the mitigations in this article exist to work around: the chunk finds the page, the document supplies the truth.

Routing keeps the cost sane. Cheap classification of the incoming question — does it name a specific document, contain an identifier, or ask a conceptual question? — decides whether to go straight to a direct lookup, run a lexical search, run a semantic search, or hand the model the full toolset for an open-ended investigation. Most production traffic is repetitive, and routing the easy 80% to a single fast path leaves the latency and token budget available for the questions that genuinely need multi-step retrieval.

What this buys you: exact lookups stay exact, conceptual questions still get semantic matching, permissions are enforced when a document is read rather than when it was indexed, the index can be rebuilt from scratch after any change to chunking or embedding models, and every answer traces back to a versioned object in blob storage that you can actually go and open.