Architecture & Codebase Exploration
The capstone is the sum of every prior module: the agent loop from Lab 02, RAG-style retrieval, memory, evals, tracing, and HITL. First the architecture and scope; then the hardest sub-problem — finding the few relevant files in a repo far too large for the context window.
Hiring research is blunt about this: an issue-to-PR coding agent is the single strongest portfolio piece you can build. It exercises everything — planning, retrieval, tool use, sandboxing, test-driven iteration, evaluation, cost accounting, and human oversight — in one artifact that a reviewer can actually run. This module builds it, then converts it into interview capital.
The six stages
- Input: a GitHub issue URL (or a local issue file) — title, body, and any repro steps.
- Explore: map the repo, locate the relevant code, and state your understanding of the bug plus a plan. Checkpoint the plan.
- Implement: write the fix in a sandboxed workspace — a git worktree or container, never the real tree.
- Verify: run the repo's test suite, write at least one new test reproducing the issue (red → green), iterate on failures up to a bounded retry cap.
- Deliver: open a draft PR (or produce a patch + PR description) — gated on HITL approval showing the diff, test results, and cost.
- Observe & evaluate: full tracing and a per-issue cost report; run across an eval set and report results.
Architecture decisions worth defending
| Decision | Options | Sensible default |
|---|---|---|
| Orchestration | Hand-rolled agent loop vs. a graph framework (e.g. LangGraph) | Checkpointed plan either way; a framework earns its keep once you need durable resume across stages |
| Plan durability | In-memory vs. persisted checkpoints | Persist — a crash mid-fix shouldn't discard exploration work |
| Sandbox | Git worktree vs. container | Worktree for speed and simplicity on trusted repos; container when running untrusted code |
| Edit strategy | Search/replace vs. full-file rewrite | Search/replace by default (cheaper, safer diffs); covered next lesson |
| Model choice | One flagship everywhere vs. cheap-explore + strong-repair split | Start with one strong generalist (claude-sonnet-5 or gpt-5.5); once traces show exploration dominates cost, route it to a cheaper model like claude-haiku-4-5. OpenAI's gpt-5.3-codex line is specialized for agentic coding and worth benchmarking for the repair stage |
Exploration strategy: agentic search over dumping
There are two ways to locate relevant code, and the agentic one usually wins for this task. Embedding-based retrieval (chunk the repo, embed, semantic search on the issue text) is fine for concept-level 'where is auth handled?' queries. But bugs are often about specific symbols, error strings, and call sites — where agentic grep-and-read shines: give the agent tools to search for symbols, list a directory, and read a file, and let it navigate the way a human engineer does. In practice you combine them: semantic search to seed candidates, then grep/read to confirm and expand.
# Colab cell — pure Python, no key needed. Builds a tiny sandbox repo on
# disk so the three tools actually run against real files.
import subprocess, pathlib
REPO = pathlib.Path("sandbox/repo").resolve() # absolute so containment checks work
REPO.mkdir(parents=True, exist_ok=True)
(REPO / "pricing.py").write_text(
"def calculate_discount(price, pct):\n"
" return price * (pct / 100)\n")
(REPO / "utils.py").write_text(
"def clamp(x, lo, hi):\n return max(lo, min(x, hi))\n")
def search_symbol(pattern: str, max_results: int = 40) -> str:
"""Grep the repo for a symbol or error string. Ripgrep if available."""
try:
out = subprocess.run(
["rg", "-n", "--max-count", "3", pattern, str(REPO)],
capture_output=True, text=True, timeout=20,
).stdout
except FileNotFoundError:
out = subprocess.run(
["grep", "-rn", pattern, str(REPO)],
capture_output=True, text=True, timeout=20,
).stdout
lines = out.splitlines()[:max_results]
return "\n".join(lines) or "no matches"
def list_dir(rel: str = ".") -> str:
target = (REPO / rel).resolve()
# Constrain to the repo — never let the agent wander the filesystem.
if REPO not in target.parents and target != REPO:
return "error: path escapes the repo"
entries = sorted(p.name + ("/" if p.is_dir() else "") for p in target.iterdir())
return "\n".join(entries)
def read_file(rel: str, start: int = 1, end: int = 400) -> str:
target = (REPO / rel).resolve()
if REPO not in target.parents and target != REPO:
return "error: path escapes the repo"
if not target.is_file():
return f"error: {rel} is not a file"
text = target.read_text(errors="replace").splitlines()
window = text[start - 1:end]
# Line numbers help the model reference and later edit precisely.
return "\n".join(f"{i + start:>5} {ln}" for i, ln in enumerate(window))
# drive the agent's three "eyes" against the sandbox repo:
print("search_symbol('calculate_discount'):\n" + search_symbol("calculate_discount"))
print("\nlist_dir('.'):\n" + list_dir("."))
print("\nread_file('pricing.py'):\n" + read_file("pricing.py"))list_dir / read_file and 'simplifies' the containment check to a plain string comparison, arguing pathlib is overkill for a sandbox that never sees untrusted input:def list_dir(rel: str = ".") -> str:
target = str((REPO / rel).resolve())
if not target.startswith(str(REPO)):
return "error: path escapes the repo"
...# Colab cell — run once. Set your key in the 🔑 panel (name it
# ANTHROPIC_API_KEY) or paste it. This is the assembled agent loop: it needs
# the tools from the previous cell wired into EXPLORE_TOOLS (schemas) and a
# run_explore_tool dispatcher, plus a real repo — it's here to read and adapt.
!pip install -q anthropic
import os
try:
from google.colab import userdata
os.environ["ANTHROPIC_API_KEY"] = userdata.get("ANTHROPIC_API_KEY")
except Exception:
from getpass import getpass
os.environ.setdefault("ANTHROPIC_API_KEY", getpass("Anthropic API key: "))
import json, pathlib
import anthropic
client = anthropic.Anthropic()
PLAN_PATH = pathlib.Path("sandbox/plan.json")
def explore_and_plan(issue_text: str) -> dict:
system = (
"You are a senior engineer triaging a bug. Use search_symbol, "
"list_dir, and read_file to locate the relevant code. Do NOT guess "
"at file contents — read them. When confident, call record_plan with "
"the files you will edit, your understanding of the bug, and the fix "
"approach. Read only what you need; the repo is large."
)
messages = [{"role": "user", "content": f"Issue:\n{issue_text}"}]
while True:
resp = client.messages.create(
model="claude-sonnet-5", max_tokens=4096,
system=system, tools=EXPLORE_TOOLS, # search/list/read/record_plan
messages=messages,
)
if resp.stop_reason != "tool_use":
continue
messages.append({"role": "assistant", "content": resp.content})
results = []
plan = None
for block in resp.content:
if block.type != "tool_use":
continue
if block.name == "record_plan":
plan = block.input # {"files":[...], "bug":"...", "fix":"..."}
else:
results.append(run_explore_tool(block))
if plan is not None:
# Checkpoint: survives a crash so W22's implement stage can resume.
PLAN_PATH.write_text(json.dumps(plan, indent=2))
return plan
messages.append({"role": "user", "content": results})Why on-demand search wins in practice
It's tempting to reach for infrastructure: chunk the repo once, embed it, keep a semantic index warm, and query it like RAG. In production coding agents this mostly lost to on-demand agentic search (grep/glob/read), for reasons that generalize past this capstone. Freshness: a repo changes every commit; an index is stale the moment someone merges, and staleness in code search is worse than in prose — a stale hit sends the agent to code that no longer exists. Keeping an index current means a background re-embed job, versioning, and a new class of bugs ('index says line 40, file says line 55'). On-demand search reads the live working tree, so it's correct by construction. Infrastructure cost: an index is a service — storage, an embedding pipeline, a latency budget, a thing that pages someone at 3am. rg is a binary that ships with the OS. For a tool a single engineer or a CI job runs, the infra tax often costs more than it saves. Model capability: frontier models got good enough at iterative, targeted search — grep an error string, read the hit, follow the import — that the coordination overhead of maintaining an index stopped paying for itself for exactly this workload: symbol- and error-string-level bug hunts, not broad conceptual questions.
| Approach | Freshness | Infra cost | Best fit |
|---|---|---|---|
| On-demand grep/glob/read | Always current — reads the live tree | None — ships with the OS | Most repos; symbol- and error-string-level bug hunts (this capstone) |
| Pre-built semantic index | Stale until re-embedded | Embedding pipeline + storage + re-index jobs | Very large/monorepo scale, or cross-repo conceptual search |
Whiteboard drills
search_symbol for a common name returns thousands of hits. What changes?"read_file tool is asked to read a 50,000-line generated file — a lockfile or a minified bundle."- ▸An issue-to-PR agent is the strongest single portfolio piece; it exercises every prior module.
- ▸Scope narrowly and honestly ('simple bug-fix issues, Python repos <10k LOC') — precise scoping signals seniority.
- ▸Six stages: input → explore → implement (sandboxed) → verify → deliver (HITL) → observe/evaluate.
- ▸The context window can't hold the repo; exploration is a retrieval problem.
- ▸Prefer agentic grep-and-read (seeded by optional semantic search) — bugs are about specific symbols and call sites.
- ▸Constrain all file paths to the repo, window reads, and checkpoint the plan so implementation can resume.