Why RAG, and the Anatomy of the Pipeline

An LLM's weights encode a snapshot of public text from training time. Ask it about your company's docs, last week's incident report, or a niche internal API and it either refuses or — worse — confidently invents. RAG (retrieval-augmented generation) sidesteps retraining entirely: at question time, search a corpus for relevant passages, paste them into the prompt as context, and instruct the model to answer only from that context, with citations. The model becomes a reasoning engine over evidence you supply, instead of an oracle recalling from memory.

Embeddings: meaning as geometry

An embedding model maps text to a fixed-length vector (hundreds to a few thousand dimensions) such that semantically similar texts land near each other. "How do I reset my password?" and "forgot login credentials" share almost no words, yet their vectors sit close together. Retrieval becomes geometry: embed the query, find the nearest chunk vectors by cosine similarity. The models that do this are bi-encoders — query and document are encoded independently, which is exactly what makes them fast (document vectors are precomputed once) and slightly blunt (the model never sees query and document together — that's the cross-encoder's job, Lesson 4).

# Colab cell 1 — run once (downloads a small local embedding model;
# no API key needed).
!pip install -q sentence-transformers

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")   # small, fast, local, 384-dim

docs = [
    "Qdrant is a vector database written in Rust.",
    "BM25 ranks documents by lexical term overlap with the query.",
    "To reset your password, open Settings and choose Security.",
]
doc_vecs = model.encode(docs, normalize_embeddings=True)

query = "I forgot my login credentials"
q_vec = model.encode(query, normalize_embeddings=True)

scores = doc_vecs @ q_vec        # dot product of unit vectors = cosine similarity
for score, doc in sorted(zip(scores, docs), reverse=True):
    print(f"{score:.3f}  {doc}")

# Colab cell 2 — run cell 1 first (it defines model, docs, doc_vecs).
# Set your key in the 🔑 panel (name it ANTHROPIC_API_KEY) or paste it.
!pip install -q anthropic

import os
try:
    from google.colab import userdata
    os.environ["ANTHROPIC_API_KEY"] = userdata.get("ANTHROPIC_API_KEY")
except Exception:
    from getpass import getpass
    os.environ.setdefault("ANTHROPIC_API_KEY", getpass("Anthropic API key: "))

import anthropic

llm = anthropic.Anthropic()

def retrieve(query: str, k: int = 3) -> list[str]:
    q = model.encode(query, normalize_embeddings=True)
    top = np.argsort(doc_vecs @ q)[::-1][:k]
    return [docs[i] for i in top]

def answer(query: str) -> str:
    chunks = retrieve(query)
    context = "\n\n".join(f"[{i + 1}] {c}" for i, c in enumerate(chunks))
    prompt = (
        "Answer the question using ONLY the numbered context below. "
        "Cite chunks like [1]. If the context does not contain the answer, "
        "say 'The corpus does not cover this.' Do not use outside knowledge.\n\n"
        f"Context:\n{context}\n\nQuestion: {query}"
    )
    resp = llm.messages.create(
        model="claude-sonnet-5", max_tokens=512,
        messages=[{"role": "user", "content": prompt}],
    )
    return next(b.text for b in resp.content if b.type == "text")

print(answer("How do I recover my account?"))

RAG vs. the alternatives: the question every interview opens with

Before any pipeline detail, expect: "models have million-token context windows now — why not just paste the whole corpus in?" and "why not fine-tune instead?" These deserve crisp answers. Against context stuffing: (1) cost — you'd re-send and re-process the corpus on every query (Module 1's quadratic lesson at corpus scale; caching helps but a 500K-token cached prefix still costs real money per read, and any corpus update invalidates it); (2) attention — retrieval quality inside a stuffed window degrades, with mid-context evidence most at risk, whereas RAG hands the model 5 pre-vetted passages; (3) access control — RAG filters at retrieval time so user A never gets user B's documents in-prompt; a stuffed context is all-or-nothing; (4) scale — corpora outgrow any window. When the 'corpus' is genuinely one document that fits comfortably, though, skipping RAG is the senior answer — no pipeline beats no pipeline.

Against fine-tuning: tuning teaches behavior — style, format, domain vocabulary, tool-use patterns — but is a poor store of facts: it's slow to update (retrain per docs change vs. re-index one file), unauditable (no citation possible — the fact is smeared across weights), and prone to making confident hallucination worse, since the model now sounds native in your domain. The clean division to say out loud: fine-tune for how the model should act, RAG for what it should know. They compose — a tuned model with retrieval — but knowledge freshness, provenance, and per-user permissions all live on the RAG side.

Whiteboard drills