Beyond Text: Images, PDFs & Files

Everything in this module so far sent content as a string. The full truth: content is a list of typed blocks, and text is only one block type. Add an image or document block and the same stateless, resend-the-array machinery now carries screenshots and PDFs. No new endpoint, no new mental model.

Images: cost and placement

import base64

img_b64 = base64.standard_b64encode(open("error.png", "rb").read()).decode()

resp = client.messages.create(
    model="claude-sonnet-5", max_tokens=1024,
    messages=[{
        "role": "user",
        "content": [
            {"type": "image",
             "source": {"type": "base64",
                        "media_type": "image/png", "data": img_b64}},
            # or, if it's already hosted:
            # {"type": "image", "source": {"type": "url", "url": "https://..."}},
            {"type": "text", "text": "What's the error in this screenshot, and the likely fix?"},
        ],
    }],
)

Do the image-token math out loud when asked — it's a resolution question, not a flat fee. Claude tiles an image into 28×28-pixel patches and bills one visual token per patch (⌈width/28⌉ × ⌈height/28⌉), so cost scales directly with pixel count — the table below shows what that looks like at real sizes. At $3/MTok even the expensive end is under 2¢ per image, but three things compound it: images ride in the history and get re-billed every turn of the conversation, they eat context-window budget, and at 10K images/day the gap between the cheap and expensive rows is real money. The lever is client-side resizing: downsample to the smallest resolution that preserves what the model needs (reading a screenshot's error text needs far less than reading a dense schematic), and pull the image out of history once it's been discussed.

OpenAI counts image tokens two different ways, and knowing which family a model is in is the whole trick:

Patch-based — the 32×32-pixel patch method, now in two tiers. The efficiency models (gpt-5-mini/-nano, gpt-5.4-mini/-nano, gpt-4.1-mini/-nano, o4-mini) count ⌈w/32⌉ × ⌈h/32⌉ patches, cap at 1,536 (proportional downscale when over), then multiply by a per-model factor — 1.62× mini · 2.46× nano · 1.72× o4-mini. The newest flagships (gpt-5.4, gpt-5.5, gpt-5.6) use the same patches but a far larger, detail-controlled budget: high allows up to 2,500 patches (2048px max side), original up to 10,000 (6000px, no forced resize) — and on 5.5 / 5.6 the default is original. The fixed 1.62 / 2.46 multipliers are a mini/nano thing; flagship image tokens track the resized patch count under whichever cap detail selects.

Tile-based — the base gpt-5, gpt-4o, gpt-4.1, and the o1/o3 line (note the split: base gpt-5 is tile-based, but the gpt-5.4+ flagships above moved to patches). Shrink the image to fit a 2048×2048 box, scale the shortest side to 768px, count 512×512 tiles, then bill base + tiles × per-tile. The lever this family hands you that Claude doesn't: detail: low pays only the flat base and skips tiles entirely — one cheap fixed cost per image when you don't need fine detail.

PDFs: text and layout together

PDFs work the same way with a document block — the model sees both the text layer and the rendered pages, so tables, stamps, and layout survive. This is the workhorse for extraction jobs: invoices, contracts, reports.

pdf_b64 = base64.standard_b64encode(open("invoice.pdf", "rb").read()).decode()

resp = client.messages.create(
    model="claude-sonnet-5", max_tokens=2048,
    output_config={"format": {"type": "json_schema", "schema": {
        "type": "object",
        "properties": {
            "line_items": {"type": "array", "items": {
                "type": "object",
                "properties": {
                    "description": {"type": "string"},
                    "qty": {"type": "integer"},
                    "unit_price": {"type": "number"},
                },
                "required": ["description", "qty", "unit_price"],
                "additionalProperties": False,
            }},
        },
        "required": ["line_items"],
        "additionalProperties": False,
    }}},
    messages=[{
        "role": "user",
        "content": [
            {"type": "document",
             "source": {"type": "base64",
                        "media_type": "application/pdf", "data": pdf_b64}},
            {"type": "text", "text": "Extract every line item."},
        ],
    }],
)

The Files API: upload once, reuse everywhere

Inlining base64 is fine for one-shot calls, but an agent that consults the same 200-page handbook every session would resend megabytes per call. The Files API fixes that: upload once, get a file_id, reference it by id in any later request.

# upload once (beta header required at the time of writing)
f = client.beta.files.upload(file=open("handbook.pdf", "rb"))

# reference by id in any later request — no re-upload, no base64
resp = client.beta.messages.create(
    model="claude-sonnet-5", max_tokens=1024,
    betas=["files-api-2025-04-14"],
    messages=[{
        "role": "user",
        "content": [
            {"type": "document", "source": {"type": "file", "file_id": f.id}},
            {"type": "text", "text": "What does the vacation policy say?"},
        ],
    }],
)

Whiteboard drills