Sending images and documents

Modern models are multimodal: a user turn can carry pictures and PDFs, not just text. This is how you build "read this receipt", "what is wrong in this screenshot", or "pull the totals out of this invoice". The message shape you already know barely changes: content becomes a list of blocks, and alongside your text blocks you add image or document blocks.

An image block carries the bytes as base64, tagged with its media type:

{"type": "image",
 "source": {"type": "base64", "media_type": "image/png", "data": B64_STRING}}

A PDF rides in a document block with media type application/pdf. You interleave blocks freely, so a single user turn can be "here is the image, now answer this":

{"role": "user", "content": [
    {"type": "image", "source": {"type": "base64", "media_type": "image/jpeg", "data": B64}},
    {"type": "text", "text": "What store issued this receipt?"}
]}

When does vision beat OCR? Classic OCR turns pixels into a flat stream of characters and loses the layout. A vision model reads the pixels and reasons about structure: it can tell a total from a subtotal, follow a two-column form, or read a handwritten note, all in one step. The tradeoff is cost and determinism: images are many tokens, and you still want your own code to validate whatever comes back. The pattern for the rest of your career: the model reads the pixels; your deterministic code still guarantees the shape.

You will build the request constructor: validate the images, order the blocks, and assemble the exact user turn.