Errors, Rate Limits & Cost Control

The failure taxonomy

import random, time
import anthropic

RETRYABLE = (anthropic.RateLimitError, anthropic.OverloadedError,
             anthropic.InternalServerError, anthropic.APIConnectionError,
             anthropic.APITimeoutError)

def call_with_retries(fn, max_retries: int = 3, base: float = 1.0):
    for attempt in range(max_retries + 1):
        try:
            return fn()
        except (anthropic.BadRequestError, anthropic.AuthenticationError,
                 anthropic.PermissionDeniedError):
            raise                       # 400/401/403 = your bug or creds. Never retry.
        except RETRYABLE as e:
            if attempt == max_retries:
                raise
            # exponential: 1s, 2s, 4s… + full jitter to avoid thundering herd
            delay = base * (2 ** attempt) * (0.5 + random.random())
            print(f"retryable error ({type(e).__name__}), "
                  f"sleeping {delay:.1f}s (attempt {attempt + 1})")
            time.sleep(delay)

Prompt caching — the agent cost lever

Agents resend a large, mostly-identical prefix every turn: system prompt, tool schemas, early conversation. Prompt caching lets the provider reuse the processed prefix — cached input tokens cost a fraction of fresh ones (Anthropic: cache reads are ~90% cheaper; writes cost a small premium) and process faster. For a 20-turn agent session whose prefix dominates, caching routinely cuts input cost by 70–90%. Two mechanics worth memorizing: caching keys on an exact prefix match, and there's a minimum cacheable prefix (roughly 1K–4K tokens depending on model) — short prompts silently don't cache at all, with no error.

resp = client.messages.create(
    model="claude-sonnet-5", max_tokens=1024,
    system=[{
        "type": "text",
        "text": LONG_SYSTEM_PROMPT,          # stable across turns
        "cache_control": {"type": "ephemeral"},   # <- cache up to here
    }],
    tools=TOOLS,                             # stable too — order matters
    messages=messages,
)
print(resp.usage.cache_read_input_tokens,    # cheap
      resp.usage.cache_creation_input_tokens)  # small premium, first call

The economics have three numbers worth knowing cold. Reads cost ~0.1× base input. Writes cost a premium: ~1.25× for the default 5-minute TTL, ~2× for the optional 1-hour TTL. So the 5-minute cache breaks even on the second request (1.25 + 0.1 < 2×), while the 1-hour cache needs about three (2 + 0.1 + 0.1 < 3×) — reach for the long TTL only when traffic is bursty with gaps longer than 5 minutes; steady traffic keeps the short cache warm for free, because every read refreshes it. Two more mechanics: you get at most 4 cache breakpoints per request, and invalidation is tiered — editing a message leaves the tools+system cache intact, but changing the tool list or the model invalidates everything, because tools render at position zero.

Choosing the model: the biggest cost lever of all

Backoff and caching shave percentages; model choice changes cost by an order of magnitude. Providers ship tiers with roughly 5–25× price spreads between the smallest and largest, and most production systems route: a cheap fast model for classification, routing, and extraction; a mid-tier workhorse for the agent loop; the flagship only for planning and the hardest reasoning.

Every provider has an equivalent ladder (OpenAI's mini/full split around gpt-5.5, etc.), and prices change — pull current numbers from the pricing page, never from memory or a course. Two routing patterns to know: static routing (each pipeline stage is assigned a tier at design time) and the cascade (try the cheap model, escalate to the expensive one only when confidence is low or validation fails). Both show up constantly in system-design interviews.

The Batch API: 50% off anything that can wait

The third cost lever, and the one most candidates forget exists: if a workload doesn't need an answer now, don't send it through the real-time endpoint at all. The Batch API takes up to ~100K requests in one submission, processes them asynchronously (most batches finish within an hour; 24 hours is the ceiling), and charges 50% of standard price on all tokens — stacking with prompt caching and cheap-tier routing. Nightly classification runs, backfills, eval suites, document-extraction pipelines: all batch-shaped.

batch = client.messages.batches.create(requests=[
    {"custom_id": f"ticket-{t.id}",
     "params": {"model": "claude-haiku-4-5", "max_tokens": 256,
                "messages": [{"role": "user",
                              "content": f"Classify: {t.text}"}]}}
    for t in tickets
])

while True:
    b = client.messages.batches.retrieve(batch.id)
    if b.processing_status == "ended":
        break
    time.sleep(60)

results = {}
for r in client.messages.batches.results(batch.id):
    if r.result.type == "succeeded":
        results[r.custom_id] = r.result.message
    else:
        log_failure(r.custom_id, r.result)   # errored | canceled | expired

Observability: the log line that answers every incident

When the bill spikes or quality drops, the difference between a 10-minute diagnosis and a lost week is whether you logged the right fields per API call from day one. The canonical structured log line for an LLM call:

• Identity: your request/session/user ids, plus the provider's request id (from the response headers — it's what support asks for).
• What ran: model, prompt/template version, tool names requested.
• Tokens by class: input_tokens, output_tokens, cache_read_input_tokens, cache_creation_input_tokens — the split matters; total input alone can't tell you the cache stopped working.
• Outcome: stop_reason, error type if any, retry count.
• Latency: total and time-to-first-token.
• Money: computed cost from a rates table you can update.

Then aggregate and alert on the derivatives: cache hit-rate dropping (a deploy broke the prefix), refusal or max_tokens rates spiking (prompt or cap regression), cost-per-session drifting up (history bloat), p95 iterations-per-turn climbing (the model is struggling with a tool). One more production discipline while you're here: retries plus side effects require idempotency. A timeout doesn't tell you whether the provider processed the request; and your tool executor's retries can re-run a charge_customer call. Idempotency keys on every side-effectful downstream call — derived from the tool_use_id — make retries safe.

def call_with_retries(fn, max_retries=8):
    for attempt in range(max_retries + 1):
        try:
            return fn()
        except anthropic.APIError:
            time.sleep(2 ** attempt)
    raise RuntimeError("gave up")

Whiteboard drills