Controlling Generation: Sampling, Thinking & Streaming

At each step of generation, the model produces a probability distribution over its entire vocabulary for the next token. Sampling parameters shape how a token gets picked from that distribution. Before the knobs make sense, walk the pipeline that runs at every single step.

Step by step: how one token gets picked

The model's last layer doesn't output a token, and it doesn't output a probability either. It outputs one raw, unbounded real number per vocabulary token — a logit. 4.2 for "Paris", -0.6 for "France". Logits can be negative, they don't sum to anything meaningful, and a logit of 4.2 on its own tells you nothing except that it's bigger than the others. They're a ranking, not a distribution — yet.

Softmax is the function that turns that vector of logits into an actual probability distribution: exponentiate every logit (which makes everything positive), then divide each by the sum of all the exponentials, so every value lands between 0 and 1 and the whole vocabulary sums to exactly 100%.

P(token_i) = exp(logit_i) / Σⱼ exp(logit_j)

The knobs: temperature and top_p

• temperature: scales the logits before softmax (divides them, technically — the animation above uses T=0.7). Near 0 → almost always the top token (near-deterministic, but not perfectly — GPU nondeterminism and ties remain). High → more diverse, more creative, more wrong. Ranges vary by provider (0–2 on OpenAI; 0–1 on Anthropic models that still accept it).
• top_p (nucleus sampling): sample only from the smallest set of tokens whose cumulative probability ≥ p. top_p=0.9 ignores the long tail entirely.
• Adjust one, not both. They interact multiplicatively and become impossible to reason about together. Pick temperature as your primary dial.
• max_tokens is a hard output cap, not a target — the model doesn't know it exists. Set it as a safety rail against runaway generation and check stop_reason for max_tokens to detect truncation.

For most of the industry — every older Claude model, every OpenAI chat model, and essentially every open-weights model you'll self-host — these dials are still live, and you set them as plain fields on the request. This is the API surface you'll meet in the overwhelming majority of existing code and tutorials, so recognize it on sight:

resp = client.messages.create(
    model="claude-sonnet-4-5",          # a Claude 4.5-or-earlier model still accepts these
    max_tokens=1024,                    # hard output cap — a safety rail, not a target
    temperature=0.2,                    # 0–1 on Anthropic; low = precise, high = varied
    # top_p=0.9,                        # nucleus sampling — tune this OR temperature, never both
    # top_k=40,                         # optional: only ever consider the 40 likeliest tokens
    stop_sequences=["\n\nHuman:"],      # end generation at any of these strings
    messages=[{"role": "user", "content": "Draft a product tagline."}],
)
print(resp.content[0].text)

The 2026 twist: frontier models removed the dials

Anthropic's Claude Opus 4.7 and 4.8 reject temperature, top_p, and top_k outright — including any one of them in a request returns a 400, even the model's own default value. Claude Sonnet 5 is a notch more lenient: omitting the parameter, or passing its default, is accepted; only a non-default value 400s. All three also drop the old fixed-budget shape — thinking: {"type": "enabled", "budget_tokens": N} now 400s — leaving a single on-state: thinking: {"type": "adaptive"}.

The control surface moved up a level: instead of shaping the token distribution yourself, you tell the model how hard to work. Adaptive thinking lets the model decide when and how much to reason before answering; effort — set inside output_config, defaulting to "high" — scales how much total work (thinking, tool calls, output) it spends, from low up to max. One default worth knowing: leave thinking out of the request entirely and Sonnet 5 runs adaptive automatically, while Opus 4.7/4.8 run with no thinking at all — set thinking: {"type": "adaptive"} explicitly if you want reasoning on the Opus tier.

OpenAI's reasoning models followed the same path from their own starting point. o1, o3, o4-mini, and the GPT-5 series reject temperature, top_p, presence_penalty, frequency_penalty, and a few other sampling-adjacent fields once reasoning is active — the API error is explicit: "Unsupported value: 'temperature' does not support 0.2 with this model. Only the default (1) value is supported." In their place, reasoning_effort (Chat Completions) or reasoning: {"effort": ...} (Responses API) controls how much internal reasoning happens, with values from "minimal"/"none" up to "high"/"xhigh" depending on model version. Unlike Anthropic's blanket removal, OpenAI's rule isn't fixed per model family — some later GPT-5.x point releases (e.g. GPT-5.2 defaulting to reasoning_effort: "none") reintroduced temperature support once reasoning is explicitly dialed down, so treat 'does this model accept temperature' as a per-version question you check in the docs, not a permanent per-family rule.

resp = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=16000,
    thinking={"type": "adaptive"},        # model decides when/how much to reason
    output_config={"effort": "medium"},   # low | medium | high | xhigh | max (default: "high")
    messages=[{"role": "user", "content": "Plan the refactor step by step."}],
)

for block in resp.content:
    if block.type == "thinking":
        pass          # empty by default (display="omitted") — pass display="summarized" for a readable summary
    elif block.type == "text":
        print(block.text)

Every response tells you why it stopped

Every provider tells you why generation ended, and production code switches on that field before reading content — most silent agent failures trace back to code that assumed the happy path. But the field name, its possible values, and even which object it lives on are provider-specific and API-specific. Anthropic has one field on one API; OpenAI has two different mechanisms depending on which API you call. Learn both — interviewers use this as a completeness check when you whiteboard a loop.

Streaming

Without streaming, the client sends one request and blocks until the model has generated the entire completion — for a long answer that's many seconds of dead air. Streaming flips this: the API holds the HTTP response open and pushes tokens to the client as they're produced, over Server-Sent Events (SSE). Time-to-first-token, not time-to-full-response, becomes the latency the user actually feels — often a 10× improvement in perceived speed for the same total generation time.

Why SSE — not a plain response, not a WebSocket

SSE is a W3C standard (part of the HTML spec) for server-to-client streaming over an ordinary HTTP connection. The server replies with Content-Type: text/event-stream and, instead of writing one body and closing, holds the connection open and emits a sequence of UTF-8 text events — each an event:/data: block terminated by a blank line — until it's finished. It's deliberately minimal: one direction (server → client), text only, carried by the same HTTP request you already made.

Contrast that with a plain HTTP request, which is all-or-nothing: the client blocks until the full response body has arrived, then processes it. Even when chunked transfer encoding moves the bytes in pieces underneath, an ordinary client hands your code the body only once it's complete — so you're back to dead air. SSE is that same streamed HTTP response plus a thin framing protocol (typed events with explicit boundaries), which is exactly what lets the client parse and render each token the moment it lands instead of waiting for the last byte.

A WebSocket starts as an HTTP request but then performs an Upgrade handshake that switches the connection to a separate, full-duplex protocol (ws://) where either side can send at any time. That bidirectionality is precisely what token streaming doesn't need: the interaction is send-one-prompt, stream-one-response. Reaching for WebSockets here buys you connection state, heartbeats, and reconnection logic to manage — plus infrastructure (corporate proxies, CDNs, load balancers) that frequently mishandles the protocol upgrade. SSE's unidirectional shape matches the request-then-response-stream shape of an API call, and because it stays plain HTTP, your bearer-token auth, proxies, retries, and request logging all keep working unchanged. That fit is why every major LLM provider streams over SSE rather than WebSockets.

with client.messages.stream(
    model="claude-sonnet-5", max_tokens=1024,
    messages=[{"role": "user", "content": "Explain SSE in one paragraph."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

final = stream.get_final_message()          # full message + usage

What SSE actually looks like on the wire

Seniors get asked to describe the event protocol, not just call the helper. The response is a long-lived HTTP response with Content-Type: text/event-stream; each event is a typed frame. The lifecycle: one message_start, then for each content block a content_block_start → many content_block_delta → content_block_stop, then message_delta (carrying the final stop_reason and usage) and message_stop.

event: message_start
data: {"type":"message_start","message":{"id":"msg_...","usage":{...}}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hel"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"lo"}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"output_tokens":12}}

event: message_stop
data: {"type":"message_stop"}

Streaming + tool calls: accumulating partial JSON

Streaming text is easy: every text_delta is a finished piece of string — you print it and move on. Tool calls are the exception, and this is the one part of streaming people get wrong. When the model decides to call a tool, the tool's arguments are a JSON object — but that object does not arrive whole. The API types it out the same way it types out prose: as a run of tiny character fragments, one per input_json_delta frame. Any single fragment is just raw characters of half-written JSON, so on its own it almost never parses.

The tool call the model wants:
    {"city": "Paris", "units": "celsius"}

How it actually arrives — one input_json_delta at a time:
    delta 1  →  {"ci
    delta 2  →  ty": "Par
    delta 3  →  is", "un
    delta 4  →  its": "celsius"}

json.loads('{"ci')                     ✗  crashes — not valid JSON yet
parse after every delta                ✗  keeps crashing until the last piece
json.loads(all fragments joined)       ✓  {"city": "Paris", "units": "celsius"}

So the key point is a two-part rule. (1) Don't parse as you go — accumulate each fragment into a string buffer and call json.loads exactly once, when content_block_stop signals the object is complete. (2) Keep one buffer per block, not one global buffer — a single response can request several tools at once, and each is a separate content block with its own index; their fragments interleave on the wire, so you key buffers by index to keep them from mixing. That's the whole trick, and it's a classic live-coding trap.

import json

tool_calls = {}   # block index -> {"name": ..., "id": ..., "buf": ...}

with client.messages.stream(model="claude-sonnet-5", max_tokens=1024,
                            tools=TOOLS, messages=messages) as stream:
    for event in stream:
        if event.type == "content_block_start" and \
                event.content_block.type == "tool_use":
            tool_calls[event.index] = {"name": event.content_block.name,
                                       "id": event.content_block.id, "buf": ""}
        elif event.type == "content_block_delta":
            if event.delta.type == "input_json_delta":
                tool_calls[event.index]["buf"] += event.delta.partial_json
            elif event.delta.type == "text_delta":
                print(event.delta.text, end="", flush=True)
        elif event.type == "content_block_stop" and event.index in tool_calls:
            call = tool_calls[event.index]
            call["input"] = json.loads(call["buf"])   # NOW it's parseable

final = stream.get_final_message()   # or just use this — the SDK accumulated it

Whiteboard drills