What’s Prompt caching?
Prompt caching lets an API skip re-processing the parts of a request that haven't changed — the system prompt, tool definitions, a long document — by storing the model's internal representation of that prefix and reusing it on the next call. It's why a long, stable system prompt can cost a fraction on the second and subsequent turns: you pay full price once, then a steep discount for cache reads.
The catch is that caching has usually been all-or-nothing past the point where a request diverges from the cached prefix: change one word in a tool definition and the whole downstream cache invalidates, even if the content that matters for this turn didn't really change. That's been a real cost for anything that mutates tools or context mid-conversation.
What's changing: Anthropic's new inline-tools beta header lets you add or redefine a tool inside a mid-conversation system message without touching the original tools array, so the cache upstream of it survives. OpenAI's new caching diagnostics do something adjacent — explicit breakpoints and hit-rate visibility, so you can see exactly where your own requests are quietly paying full price.
The debate underneath this is whether caching should stay an implementation detail the provider manages, or become a first-class thing developers design against — the way you'd design around a database index boundary. The direction both labs are moving suggests the latter is winning.