Of all the AI cost optimizations available today, prompt caching has the best ratio of impact to implementation effort, and it remains surprisingly under-adopted. Teams that skip it are often paying full price repeatedly for content that has not changed between calls, for no reason other than not having wired up a feature that already exists in the API they are using.
What prompt caching actually does
When you send a request to a model, a portion of that prompt is often identical or near-identical to a previous request — a long system prompt, a reference document, a set of few-shot examples. Prompt caching lets the provider store the processed representation of that static content after the first call, so subsequent calls that reuse the same prefix pay a small fraction of the normal cost for that cached portion, rather than paying full price to reprocess it every single time.
How it works across providers
Anthropic's implementation uses an explicit cache_control marker in the request, letting you specify exactly which portion of the prompt should be cached, with a defined time-to-live before the cache expires and needs to be rewritten. This gives you precise control but requires deliberately structuring your prompt and marking the cache boundary yourself.
OpenAI's approach leans more toward automatic prefix caching, where the system detects repeated prompt prefixes across requests and applies the discount without requiring an explicit marker in most cases, trading some of Anthropic's fine-grained control for simpler adoption with less code change required.
How much it actually saves
Cached tokens typically cost around a tenth of the normal input token price, which means for a prompt where the bulk of the content is static context and only a small portion changes per call, the cost reduction on that static portion approaches 90%. For applications making frequent calls with a long, mostly-unchanging system prompt or document context — a chatbot with an extensive system prompt, a document Q&A feature referencing the same source material repeatedly, a coding agent with a large repository context loaded — this adds up to a substantial fraction of total API spend.
The patterns that benefit most
Long system prompts used across many requests are close to a textbook case — write the system prompt once, mark it for caching, and every subsequent call in that session or that day pays the discounted rate for that portion. Document context for RAG-style applications where the same reference material gets included across multiple related queries benefits enormously, particularly for features like document Q&A where a user asks several questions about the same uploaded file in one session. Few-shot examples included in a prompt to guide output format or style are typically static across many calls and are an easy target for caching.
How to implement it without over-engineering
Start by identifying the largest static block in your most frequently called prompts — usually the system prompt or a reference document — and mark that specific block for caching rather than trying to cache the entire prompt including the parts that genuinely change every call. Caching only pays off for content that is actually reused across multiple calls within the cache's time-to-live window, so a prompt where every part changes every time gets no benefit from marking it for caching.
Caveats worth knowing before you rely on it
Cache time-to-live windows are limited — typically minutes, not hours — so caching only helps for content reused within that window, not content called once a day. Cache hits are not guaranteed under all conditions; if the cached prefix does not match exactly, including whitespace differences, the cache misses and you pay full price for that call. And caching adds a small amount of latency variance on the very first call that populates the cache, though this is negligible against the savings on every subsequent call that hits it.
Why this is worth doing properly, not just once
Prompt caching is one of those optimizations that pays for itself many times over relative to the implementation effort, and yet it is skipped surprisingly often simply because teams do not revisit prompt structure once a feature is shipped and working. Our engineers build this in as a standard practice on AI features from day one rather than as a later optimization pass. If your AI feature spend looks higher than it should, our AI engineering teams can review your prompt structure, or contact us directly.
