Chunking: The Highest-Leverage Decision

A chunk is what gets embedded, what gets retrieved, and what the model reads as evidence. That triple duty creates tension: small chunks embed crisply (one idea per vector) but may lack the surrounding context needed to actually answer; large chunks carry context but their embeddings blur into topic soup and they burn prompt budget. And if a fact straddles a chunk boundary — question in one chunk, answer in the next — neither chunk retrieves well. This is why chunking is the highest-leverage decision in the pipeline: errors here are unrecoverable downstream.

Fixed-size vs. structural

# Colab cell 1 — pure Python: runs with no key and no installs.
def fixed_size_chunks(text: str, size: int = 350, overlap: int = 50) -> list[str]:
    """Split into word windows of ~size words, consecutive windows sharing
    'overlap' words so boundary-straddling facts survive in one piece."""
    words = text.split()
    chunks, start = [], 0
    while start < len(words):
        end = min(start + size, len(words))
        chunks.append(" ".join(words[start:end]))
        if end == len(words):
            break
        start = end - overlap          # step back to create the overlap
    return chunks

# demo: numbered words make the shared boundary regions visible
demo_text = " ".join(f"w{i}" for i in range(1, 121))   # w1 ... w120
for piece in fixed_size_chunks(demo_text, size=50, overlap=10):
    words = piece.split()
    print(f"{len(words):>3} words: {words[0]} ... {words[-1]}")

Two upgrades seniors are expected to know

Contextual retrieval attacks the core weakness of chunks: they're read out of context. A chunk saying "set this flag to true and restart the service" embeds — and reads — poorly because nothing says which flag or service. The heading + text trick above is the free version; the full version has an LLM write a 1–2 sentence situating blurb per chunk at index time ("This passage is from the Acme Gateway operations guide, section on retry configuration..."), prepended before embedding and indexing. It costs one cheap LLM call per chunk once, offline — the good side of the cost asymmetry, since the index is built once and queried forever — and it substantially cuts retrieval failures on corpora where chunks are elliptical. Batch API + prompt caching (the document rides in the cached prefix while each chunk varies) make it cheap at scale — Module 1's cost levers composing.

Small-to-big (parent-document) retrieval resolves the size tension by refusing to choose: embed small, return big. Index sentence- or paragraph-sized units for crisp matching, but store a pointer from each to its parent section; at query time, match on the small unit, then hand the parent to the generator. You get precise vectors and sufficient evidence context. Costs to name: a two-level store, dedup when several small units share a parent, and a larger generation prompt. The umbrella idea behind both techniques — and the phrase worth saying in an interview — is decoupling the retrieval representation from the generation payload: what you match on and what the model reads no longer have to be the same bytes.

# Colab cell 2 — run cell 1 first (it defines fixed_size_chunks).
import re

def structural_chunks(md: str, doc_id: str,
                      max_words: int = 350, overlap: int = 50) -> list[dict]:
    """Split on headings first; size-cap oversized sections with the
    fixed-size chunker. Keep the heading path for citations, and prepend
    it to the text we embed so section context reaches the vector."""
    parts = re.split(r"(?m)^(#{1,4}\s.*)$", md)
    chunks, heading = [], "(intro)"
    for part in parts:
        part = part.strip()
        if not part:
            continue
        if re.match(r"^#{1,4}\s", part):
            heading = part.lstrip("#").strip()
            continue
        for i, piece in enumerate(fixed_size_chunks(part, max_words, overlap)):
            chunks.append({
                "doc_id": doc_id,
                "heading": heading,
                "position": i,
                "text": piece,
                "embed_text": f"{heading}\n{piece}",   # heading rides into the embedding
            })
    return chunks

SAMPLE_MD = """# Acme Gateway runbook

## Retry configuration
Set retry_backoff_max to cap exponential backoff. Workers retry failed
jobs up to five times before dead-lettering.

## Connection errors
ERR_CONN_5031 means the gateway dropped a keep-alive connection.
Restart the connection pool or raise the idle timeout.
"""

for c in structural_chunks(SAMPLE_MD, doc_id="runbook"):
    print(f"[{c['doc_id']} / {c['heading']} / #{c['position']}] {c['text'][:48]}...")

• Chunk size: start around 250–500 words for technical docs. Smaller for FAQ-like corpora (one Q&A per chunk), larger for narrative prose.
• Overlap: 10–20% of chunk size. More than that mostly buys you duplicate retrievals.
• Never mix units across the corpus without recording which chunker produced each chunk — you can't A/B what you can't attribute.
• Chunking is an eval-set question, not a taste question: re-run retrieval metrics (Lesson 5) for each candidate strategy and let precision@5 decide.

Whiteboard drills