Prompt Caching & Stable-Prefix Ordering

How Prompt Caching Works

When a large, stable prefix -- a system prompt, a policy document, a few-shot example block -- repeats across many requests, Prompt Caching lets that prefix be reused instead of reprocessed from scratch each time:

• Cache reads are cheap
• Cache writes carry a slight premium

The economics pay off with reuse: the more requests share the identical stable prefix, the more the cheap reads outweigh the one-time write premium.

The Concrete Numbers Behind "Cheap Reads, Slight Premium"

The qualitative framing above ("reads are cheap, writes carry a premium") is the durable, exam-relevant idea. But an architect building an actual cost model needs the approximate magnitudes, not just the direction, because the size of the effect determines whether caching is worth the added system complexity for a given workload. As illustrative orders of magnitude (verify exact current rates against live Anthropic pricing documentation before finalizing a cost model, since pricing terms are exactly the kind of detail that changes over time):

The practical consequence of this table is a break-even calculation an architect should actually run, not skip: because the first request pays a WRITE premium (more expensive than uncached) and only later requests get the discounted READ rate, a prefix that gets reused only once or twice within the TTL window may not save money at all -- the one expensive write can outweigh a small number of cheap reads. Caching pays off specifically when a stable prefix is reused often enough, and frequently enough within the TTL window, that the accumulated read discount overwhelms the one-time write premium. A high-traffic production endpoint with a stable system prompt reused thousands of times a minute is an obvious win; a background job that fires the same prompt twice a day, five minutes apart is not automatically a win and deserves the actual arithmetic before you assume caching helps.

This is also why the TTL matters architecturally, not just as a technical footnote: if your traffic pattern has gaps longer than the TTL between reuses of the same prefix, the cache entry has already expired by the time the next request arrives, and that request pays the write premium again instead of getting a read discount. Bursty, low-frequency traffic against a long stable prompt can end up paying write premiums repeatedly without ever accumulating enough reads inside a single TTL window to profit from the mechanism -- which is a legitimate reason a design might choose NOT to lean on caching for a specific low-frequency path, even though the prefix itself is perfectly stable and would otherwise be an ideal caching candidate.

The Ordering Principle

Order stable content first, dynamic content last, so the cacheable prefix is as long as possible. This is a direct architectural consequence of system/user placement: the stable system prompt and policy content forms the prefix; the varying user message comes after it. A longer, uninterrupted stable prefix means a bigger cache hit, which cuts both time-to-first-token and per-request cost.

Common exam traps

• Putting dynamic content before the stable prefix -- this breaks the cacheable prefix and defeats Prompt Caching entirely, since caching requires the prefix to be byte-identical across calls.
• Confusing Prompt Caching (reusing a stable prefix *within* a single model call) with retrieval (fetching external documents from a knowledge base). They solve different problems: caching is a reuse/cost optimization for content you're already sending; retrieval is how you decide what content to send in the first place.
• Truncating a needed policy document to save tokens instead of caching it. Truncation trades away correctness (the model no longer sees the full policy) for a cost saving that caching would have delivered without any loss of information.