Retrieval Augmented Generation (RAG), undefined

Blob Storage: The Document Layer

Blob (object) storage — Amazon S3, Google Cloud Storage, Azure Blob Storage, Cloudflare R2, or self-hosted MinIO — is where the raw bytes of your corpus belong: the original PDFs, HTML captures, transcripts, spreadsheets, and images. It is cheap per gigabyte, effectively unlimited, extremely durable, and versioned, which makes it the natural source of truth that every other component points back to.

The critical property to understand is what blob storage cannot do. It is a key-value store: you can GET a key and you can LIST a prefix. You cannot ask it a question about content. That means your key layout is your index — the folder structure, naming convention, and accompanying metadata are the entire navigational surface that a retrieval tool (or an LLM) will have to work with.

Key layout: design paths so that a prefix listing is already a meaningful filter. A layout like tenant/source/domain/year/doc-id/version/file.pdf lets you scope a search to one customer, one data source, or one year with a single LIST call, and it makes per-prefix IAM policies possible.

Metadata catalog: pair the bucket with a row-per-document table in a database you can actually query (Postgres, DynamoDB, SQLite for small corpora). Store the object key, title, source URL, author, created/updated timestamps, MIME type, size, content checksum, tags, access-control labels, and pointers to derived artifacts. Filtering, sorting, and permission checks happen here — never by scanning the bucket.

Derived artifacts: ingestion is a pipeline, and each stage should be persisted next to the original: raw/report.pdf → extracted/report.md → chunks/report.jsonl. Parsing a PDF is slow and expensive; do it once, store the result, and let every downstream consumer read the clean text.

Access and lifecycle: serve documents through short-lived presigned URLs rather than making buckets public, scope IAM credentials per prefix, enable object versioning so a re-ingest never destroys history, and use lifecycle rules to move cold originals to archival tiers while keeping the extracted text hot.

What belongs where — a division of responsibility that keeps systems recoverable:

Blob storage: the bytes. The only place the original document exists. Immutable and versioned.
Metadata catalog: the facts you filter, sort, and authorize on. Cheap to query, easy to reindex.
Vector store: embeddings plus a pointer back to the blob key and character offsets — an index, not a database of record.

The rule that saves you: never let the vector store be the only copy of anything. If you can rebuild the entire index from blob storage plus the catalog with one job, then changing your chunking strategy or your embedding model is a Tuesday afternoon. If you can't, it's a migration project.

Storing images and multimodal content: blobs handle this naturally — the image itself stays in the bucket, while the catalog holds a generated caption or description, and the vector store (if used) embeds that description. The model receives the caption during search and can fetch the actual image by key when it needs to look at it.

Implementation Example:

import boto3, hashlib, json, datetime

s3 = boto3.client("s3")
BUCKET = "acme-knowledge-base"

def object_key(tenant, source, doc_id, version, filename):
    """Layout is the index: prefix listings become meaningful filters."""
    year = datetime.date.today().year
    return f"{tenant}/{source}/{year}/{doc_id}/v{version}/{filename}"

def ingest(tenant, source, doc_id, filename, body: bytes, meta: dict):
    checksum = hashlib.sha256(body).hexdigest()
    key = object_key(tenant, source, doc_id, meta.get("version", 1), filename)

    # 1. Raw bytes -> blob storage (the source of truth)
    s3.put_object(
        Bucket=BUCKET,
        Key=key,
        Body=body,
        ContentType=meta["mime_type"],
        Metadata={"checksum": checksum, "title": meta["title"]},
    )

    # 2. Derived text -> stored beside it, so parsing happens exactly once
    text = extract_text(body, meta["mime_type"])
    s3.put_object(Bucket=BUCKET, Key=f"extracted/{doc_id}.md", Body=text.encode())

    # 3. Facts -> the catalog, which is what you actually query
    catalog.upsert({
        "doc_id": doc_id,
        "tenant": tenant,
        "source": source,
        "blob_key": key,
        "text_key": f"extracted/{doc_id}.md",
        "title": meta["title"],
        "author": meta.get("author"),
        "updated_at": datetime.datetime.utcnow().isoformat(),
        "acl": meta.get("acl", ["public"]),
        "checksum": checksum,
    })
    return doc_id

def presigned_url(blob_key, ttl=300):
    """Hand out time-limited links instead of public buckets."""
    return s3.generate_presigned_url(
        "get_object", Params={"Bucket": BUCKET, "Key": blob_key}, ExpiresIn=ttl
    )