Provider-side prompt caching
The last lesson skipped the model when a request repeated. This one makes the request itself cheaper. Most providers offer prompt caching: you mark a stable prefix of your prompt, and on the next call that reuses it the provider skips re-processing those tokens and bills them at a steep discount. It is a different lever from the response cache you just built:
- Response cache (last lesson): the whole answer is reused, so you skip the call entirely. Only works when the question repeats.
- Prompt cache (this lesson): the call still happens and you still get a fresh answer, but the shared prefix is cheaper. Works even when every question is different.
What belongs in the cached prefix? The big, stable stuff that rides along on every call: a long system prompt, tool definitions, few-shot examples, a retrieved document you will ask several questions about. You put the variable part (the actual user question) after the cached breakpoint. With Anthropic you mark the boundary with a cache_control block:
system=[{"type": "text", "text": LONG_SYSTEM_PROMPT,
"cache_control": {"type": "ephemeral"}}]The economics have three rates, all relative to the normal input price. A cache write (the first call, which stores the prefix) costs about 1.25x input. A cache read (a later call that hits the stored prefix) costs about 0.1x input. A miss is just 1x. The entry lives for a few minutes and its timer refreshes on every hit, so a busy prefix stays warm. The break-even is quick: if a big prefix is reused even a handful of times, the one-time 1.25x write is dwarfed by paying 0.1x instead of 1x on every reuse.
You will build the cost model that decides whether caching a prefix pays off.
Model the economics. Prices are dollars per 1,000,000 tokens; use 3.0 for input and 15.0 for output. (1) uncached_cost(prefix_tokens, variable_tokens, output_tokens, calls): every call pays full input price on prefix_tokens + variable_tokens plus output price on output_tokens, times calls. (2) cached_cost(prefix_tokens, variable_tokens, output_tokens, calls): the prefix is written once at 1.25x input price, then read on the remaining calls - 1 calls at 0.1x input price; the variable tokens always pay full input price, and output always pays full output price on every call. Assume calls >= 1. (3) caching_saves(prefix_tokens, variable_tokens, output_tokens, calls) returns True when cached_cost is strictly less than uncached_cost. Return plain floats (dollars).
This lesson is locked
Lessons open one at a time. Finish the previous lesson to unlock this one.