Messages Are the Only State

When you chat with Claude or ChatGPT, it feels like the model remembers you. It doesn't. Every single API call is a blank slate. The provider's server receives your request, runs the model over the tokens you sent, returns a completion, and forgets you existed. What creates the illusion of memory is that your code resends the entire conversation history on every call.

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY

messages = []  # <- this list IS the conversation. You own it.

def chat(user_text: str) -> str:
    messages.append({"role": "user", "content": user_text})
    response = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=1024,
        system="You are a concise engineering assistant.",
        messages=messages,        # full history, every single time
    )
    reply = response.content[0].text
    messages.append({"role": "assistant", "content": reply})
    return reply

print(chat("My name is Wenming."))
print(chat("What's my name?"))   # works ONLY because we resent turn 1

It's just JSON over POST

The SDK is a thin convenience wrapper. Strip it away and every call in this course is one HTTPS POST with an auth header and a JSON body. Seeing it raw once makes everything else less magical — and it's what you'd write from a language with no official SDK.

curl https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{
    "model": "claude-sonnet-5",
    "max_tokens": 1024,
    "system": "You are a concise engineering assistant.",
    "messages": [{"role": "user", "content": "My name is Wenming."}]
  }'

What the server actually does with your POST

When your request lands, the server tokenizes the entire prompt and runs prefill — one massively parallel pass over all input tokens that builds the model's internal working state (the KV cache). Then it decodes output tokens one at a time, each step conditioned on everything before it. When the response finishes, that working state is thrown away. This is why input and output tokens are priced differently — prefill parallelizes across the GPU while decoding is inherently serial — and it's why the next call starts from zero.

Statelessness isn't laziness — it's an architectural choice that buys horizontal scale. Because no server holds your conversation, any machine in the fleet can serve your next request: no session affinity, no replicated conversation store, trivial failover. The cost is pushed onto you (resend everything), then partially refunded by prompt caching (Lesson 5), which keeps a processed prefix warm so identical prefixes skip prefill. A senior answer connects these dots out loud: stateless → resend → quadratic cost → caching as the mitigation.

The roles

Tokens: the currency of everything

Models don't see characters — they see tokens, subword chunks (roughly 3–4 English characters, ~¾ of a word each). Every model has a context window: the maximum tokens of input + output it can handle in one call (hundreds of thousands of tokens on frontier models — check your model's docs, these numbers change). You pay per token, input and output priced separately, and output tokens typically cost several times more.

Here's the trap that surprises everyone: because you resend history every turn, cost grows quadratically with conversation length. Turn 10 doesn't cost one turn's tokens — it re-processes turns 1–9 as input, plus its own. A 10-turn conversation averaging 500 tokens per turn means turn 10's call alone sends ~4,500 input tokens, and the whole session has processed ~25,000 cumulative input tokens.

# Exact pre-send count — free, no generation. Pass the SAME system,
# messages, and tools you'll actually send, or the count will be low.
count = client.messages.count_tokens(
    model="claude-sonnet-5",
    system="You are a concise engineering assistant.",
    messages=messages,
    # tools=tools,   # tool schemas are input tokens too — include them
)
if count.input_tokens > 150_000:        # gate before spending on a big call
    messages = compact(messages)        # trim/summarize (Module 4)

# Then log real usage on EVERY response — you can't manage what you can't see.
resp = client.messages.create(
    model="claude-sonnet-5", max_tokens=256, messages=messages,
)
u = resp.usage
log.info("tokens in=%d out=%d cache_read=%d cache_write=%d",
         u.input_tokens, u.output_tokens,
         u.cache_read_input_tokens, u.cache_creation_input_tokens)

MAX_MSGS = 10

def chat(user_text: str) -> str:
    global messages
    messages.append({"role": "user", "content": user_text})
    messages = messages[-MAX_MSGS:]        # keep the context small
    resp = client.messages.create(
        model="claude-sonnet-5", max_tokens=1024, messages=messages,
    )
    reply = resp.content[0].text
    messages.append({"role": "assistant", "content": reply})
    return reply

Whiteboard drills

Answer each out loud before revealing — in the real loop you'll be talking while writing. The answers below are pitched at the senior bar and end with the follow-up probe an interviewer would fire next.