Sampling Parameters & Streaming

At each step of generation, the model produces a probability distribution over its entire vocabulary for the next token. Sampling parameters shape how a token gets picked from that distribution.

• temperature (0–1 Anthropic, 0–2 OpenAI): scales the logits before softmax. Near 0 → almost always the top token (near-deterministic, but not perfectly — GPU nondeterminism and ties remain). High → more diverse, more creative, more wrong.
• top_p (nucleus sampling): sample only from the smallest set of tokens whose cumulative probability ≥ p. top_p=0.9 ignores the long tail entirely.
• Adjust one, not both. They interact multiplicatively and become impossible to reason about together. Pick temperature as your primary dial.
• max_tokens is a hard output cap, not a target — the model doesn't know it exists. Set it as a safety rail against runaway generation and check stop_reason for max_tokens to detect truncation.

Streaming

Without streaming you wait for the full completion before showing anything — for a long answer that's many seconds of dead air. With stream=True the API returns server-sent events (SSE), delivering tokens as they're generated. Time-to-first-token becomes your perceived latency, which is often 10× better than time-to-full-response.

# Anthropic
with client.messages.stream(
    model="claude-sonnet-4-5", max_tokens=1024,
    messages=[{"role": "user", "content": "Explain SSE in one paragraph."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)
final = stream.get_final_message()          # full message + usage

# OpenAI
resp = openai_client.chat.completions.create(
    model="gpt-4o", stream=True,
    messages=[{"role": "user", "content": "Explain SSE in one paragraph."}],
)
for chunk in resp:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)