The Eval Pyramid

Every agent demo works. The demo is three hand-picked inputs the author already knows succeed. Production is the other ten thousand inputs, and the gap between demo and production is exactly the gap that evals close. An eval is a repeatable, automated judgment of whether your agent did the right thing on inputs it hasn't been tuned against. Without one, every prompt tweak is a coin flip you can't score.

Three tiers, cheapest first

Think of evals as a pyramid. The base is wide, cheap, and deterministic; each tier up is narrower, costlier, and more subjective. You run the base on every commit, the middle on every meaningful change, and the top on a sample. Push every judgment as far down the pyramid as it will honestly go — a check you can write as an assertion should never be handed to a model or a human.

The base: deterministic assertions

The most valuable evals are the boring ones. A huge fraction of agent quality is checkable without any model in the loop: the output parses, the right tool fired with the right arguments, a citation actually appears in the source corpus, no forbidden tool was touched, the run stayed under a token budget. These are fast, free, and never flaky — write them first and write a lot of them.

# Colab cell — pure Python + pytest, no API key needed. A stub agent
# stands in for your real one so the assertion evals actually run.
!pip install -q pytest

import json
from dataclasses import dataclass

import pytest


@dataclass
class ToolCall:
    name: str


@dataclass
class Result:
    text: str
    tool_calls: list


def run_agent(prompt: str) -> Result:
    # stub router standing in for your real agent (returns .text/.tool_calls)
    p = prompt.lower()
    if "refund" in p and "$900" in prompt:
        return Result("Escalating to a human.", [ToolCall("escalate_to_human")])
    if "refund" in p:
        return Result("Refund issued.", [ToolCall("issue_refund")])
    if "classify" in p:
        return Result('{"category": "billing"}', [])
    return Result("ok", [])


# A "golden" dataset: inputs paired with checkable expectations.
CASES = [
    {
        "id": "refund_under_limit",
        "prompt": "Refund my $40 order #A123, it arrived broken.",
        "must_call": "issue_refund",
        "must_not_call": "escalate_to_human",
    },
    {
        "id": "refund_over_limit",
        "prompt": "Refund my $900 order #B456.",
        "must_call": "escalate_to_human",
        "must_not_call": "issue_refund",
    },
]

@pytest.mark.parametrize("case", CASES, ids=lambda c: c["id"])
def test_tool_selection(case):
    result = run_agent(case["prompt"])
    called = {c.name for c in result.tool_calls}

    # Deterministic: exact tool-routing behavior, no model judgment needed.
    assert case["must_call"] in called, (
        f"expected {case['must_call']}, got {called}"
    )
    assert case["must_not_call"] not in called, (
        f"forbidden tool {case['must_not_call']} was called"
    )

def test_output_is_valid_json_when_structured():
    result = run_agent("Classify ticket: my card was double charged.")
    parsed = json.loads(result.text)          # raises if malformed
    assert parsed["category"] in {"billing", "bug", "feature_request", "other"}


# pytest discovers these automatically (`pytest -q`); here we call them
# directly so the cell prints a green result:
for case in CASES:
    test_tool_selection(case)
    print(f"PASS  test_tool_selection[{case['id']}]")
test_output_is_valid_json_when_structured()
print("PASS  test_output_is_valid_json_when_structured")

Task success vs. per-step correctness

Track two different numbers and never conflate them. Task success rate asks: did the whole run achieve the user's goal? Per-step correctness asks: at each turn, was the tool choice and argument right? A high task-success rate can hide a swamp of wrong first steps the agent recovered from — expensive, slow recoveries that will break the moment the environment shifts. A high per-step rate with low task success means the pieces are right but the orchestration is wrong. You need both to know where to fix.

# Colab cell — pure Python, no key needed; run it as-is.
from dataclasses import dataclass

@dataclass
class RunScore:
    task_success: bool          # did the end state match the goal?
    steps_total: int
    steps_correct: int          # per-step tool+arg correctness

def summarize(scores: list[RunScore]) -> dict:
    n = len(scores)
    task_rate = sum(s.task_success for s in scores) / n
    step_num = sum(s.steps_correct for s in scores)
    step_den = sum(s.steps_total for s in scores)
    step_rate = step_num / step_den if step_den else 0.0
    return {
        "task_success_rate": round(task_rate, 3),
        "per_step_correctness": round(step_rate, 3),
        "n_runs": n,
        # A big gap here is the signal: high task, low step = lucky recovery.
        "recovery_gap": round(task_rate - step_rate, 3),
    }

print(summarize([
    RunScore(True, 4, 2),   # succeeded, but half the steps were wrong
    RunScore(True, 3, 3),
    RunScore(False, 5, 4),  # steps mostly right, task still failed
]))

Eval economics: matching cost to cadence

The three tiers don't just differ in rigor — they differ in unit cost by orders of magnitude, and cadence should track that gap. Deterministic assertions cost fractions of a cent and run in milliseconds, so run hundreds of them on every single commit. A judge call runs anywhere from a fraction of a cent to a dollar or two depending on context size, so a 50–200 case judged suite costs a few dollars and a couple of minutes — affordable on every prompt/tool/model change, but too slow and too expensive to run on every keystroke. Human review costs real labor — call it minutes to tens of minutes per item, loaded cost of dollars each — so it's sampled: a fixed percentage of live traffic reviewed weekly for freshness, plus a fuller audit on a slower cadence (many teams land on quarterly) to catch judge drift and rubric staleness. This is the cost/frequency/fidelity trade made explicit: cheaper tiers run far more often but answer a narrower question; fidelity to 'did this actually satisfy the user' rises as you go up the pyramid, and cost rises faster than fidelity does — which is exactly why you push every judgment as far down as it honestly goes before reaching for the next tier.

Capability evals vs. regression evals

The same pyramid machinery answers two different questions, and conflating them into one aggregate number is a common, costly mistake. Capability evals ask are we getting better? — run a candidate change (new model version, rewritten prompt, a new technique) against a fixed, often deliberately hard benchmark and compare the score to the current baseline; you run these when deciding whether a change is worth shipping. Regression evals ask did we break something that used to work? — run the suite of accreted past failures (Lesson 3's territory) and look for any score decrease from baseline; you run these on every change, unconditionally. A change can raise the capability score while quietly regressing a previously-fixed edge case, and a team that only watches one blended average will ship it, because the net moved up. The discipline: report the two deltas separately in every change review — capability delta where up is good, regression delta where any red is a blocker — and never let a big capability win buy forgiveness for a regression.

Whiteboard drills