5.6 Prompt Caching and the Message Batches API

When many requests share a large, stable prefix — a system prompt, a long reference document, a set of tool definitions — reprocessing that entire prefix at full price on every single call is wasteful, because its content genuinely hasn't changed between calls. Prompt caching solves this: mark the end of the stable prefix with a `cache_control` field of type `"ephemeral"` on the last content block you want cached. That field marks a **breakpoint** — everything up to and including that block is cached as a unit. You can place **up to 4 breakpoints** per request, which matters once a prompt has more than one genuinely stable segment (say, a stable tool-definition block plus a separately stable, longer reference document appended after it).

Breakpoint placement has to respect one structural fact the exam likes to test directly: the API always processes a request in a **fixed order — tools first, then the system prompt, then messages.** A breakpoint placed right after the tool definitions caches the tools alone; a breakpoint placed at the end of the system prompt caches tools AND system together, since both preceded it in that fixed order. There is no way to cache message content "ahead of" the system prompt — messages are always processed after the system prompt regardless of where a breakpoint sits.

The two rates in that structure point in opposite directions, and the exam expects the actual numbers, not just the direction. Cache writes are billed at a premium over the base input-token price: **1.25x for the default 5-minute TTL**, or **2x for the opt-in 1-hour TTL**. Cache reads cost only **0.1x** standard input price — a 90% discount. The economics only pay off through reuse *within the TTL window*: the call that writes the cache pays the premium, and every subsequent call that reads that same cached prefix before it expires pays only 0.1x. A single one-off call that writes a cache and is never read again pays that premium with nothing to offset it — that call costs strictly MORE than the same call would have cost with no caching at all.

Worked example: a 2,000-token system prompt is stable and gets reused across 50 calls, all made within a single 5-minute window (each read resets the cache's 5-minute clock, so the cache stays alive throughout a burst like this). Without caching, the cost is 50 calls x 2,000 tokens x 1.0 (base rate) = 100,000 token-units. With caching: one write at 1.25x (2,000 x 1.25 = 2,500 units) plus 49 reads at 0.1x each (49 x 2,000 x 0.1 = 9,800 units), for a total of 12,300 units — roughly **12% of the uncached cost**, an ~8x reduction, driven entirely by the 49 cheap reads absorbing the one write premium.

A single static prefix is the simple case, but plenty of real prompts grow incrementally over a session — an agent loop that keeps appending tool results to a running conversation is the classic example. Naively, you might expect that every new appended token invalidates the entire cache built up so far, forcing a full, expensive reprocessing of the whole growing context on every turn.

Cache checkpointing avoids that. It lets you cache at multiple points along a growing prompt, so that as new content is appended at the end, the earlier, unchanged portions of the prompt stay cached and cheap to reuse — only the newly appended tail needs to be processed at full price. For a long-running agent session, this is the difference between the cache cost climbing back to full price on every single turn versus staying mostly discounted throughout the session's growth.

A common but avoidable mistake is reaching for truncation as the cost-saving move on a large document or long stable prefix, when caching the full content would have preserved correctness at a similar or better cost profile. Truncating a needed reference document to save tokens trades away information the task may genuinely need, in exchange for a saving that caching could deliver without any loss of content at all.

The decision rule is straightforward: if a large piece of content is stable and will be reused across multiple calls, cache it rather than cutting it down. Truncation is a tool for content that genuinely isn't needed in full, or that has no reuse pattern to cache against — not a default response to "this is taking up a lot of tokens."

The Message Batches API processes a large set of requests asynchronously, completing within a 24-hour window, in exchange for a roughly 50% per-token discount versus standard synchronous pricing. It fits latency-tolerant, high-volume jobs — bulk classification, large-scale content generation, offline evaluation runs — where no individual request needs an immediate, synchronous response.

The defining exam point is what batching trades for that discount: speed. It is slower by design — results arrive asynchronously, potentially hours later — so it reduces cost, not latency. This puts it in a genuinely different category from the other levers in this domain. Prompt caching and right-sizing the model to a smaller tier both reduce cost AND latency. Batching only reduces cost; reaching for it on a latency-sensitive step would be a mismatch, not an optimization.

The exact limits and mechanics matter as much as the discount itself. A single batch call accepts **up to 100,000 requests OR 256 MB of total request size, whichever limit is hit first** — a batch of many small requests can hit the count ceiling well before the size ceiling, and a batch of fewer, larger requests (each carrying a big document, say) can hit the size ceiling long before 100,000 requests. Submitting a batch returns a `batch_id` immediately; the batch itself then runs asynchronously, and your code **polls** for completion — checking the batch's status on some interval until the API reports it has finished, or until the 24-hour window elapses at the latest. Once complete, you retrieve the results.

The detail most likely to trip someone up in practice: **batch results return in ARBITRARY order**, not the order the requests were submitted in. If you submit 500 requests, you cannot assume result #37 in the returned list corresponds to input #37 you sent. The fix is the **`custom_id` field**: set it on each request when you submit the batch, and that same `custom_id` is echoed back on the corresponding result. Matching results back to inputs means keying off `custom_id`, never off list position.

Batching and caching are not competing levers — they compound. A scheduled, non-urgent batch job that reuses the same long system prompt across many requests inside one batch call gets the ~50% batch discount on every request AND the caching discount on the repeated prefix inside each one. Chunking a list and looping over the *synchronous* API, one request at a time, is a common false substitute for this — it produces the same number of individual API calls as the un-chunked version, runs into the exact same rate limits, and captures none of the batch discount. The Message Batches API is a genuinely different submission model (one call, up to 100,000 requests/256 MB, a batch_id, asynchronous completion), not a smaller batch size wrapped around the synchronous endpoint.

Task Statement 5.6 questions typically describe a cost or latency problem and ask which lever — caching, checkpointing, batching, or a smaller model — fits, or ask you to distinguish cost levers from latency levers directly.

• •Assuming cache reads and writes are priced the same. ✗ An answer treating caching as uniformly discounted from the first call. ✓ The answer recognizing writes carry a slight premium and reads are the discounted, reuse-driven savings.
• •Truncating a reused stable document to save tokens. ✗ An answer that cuts down a document that will be reused across calls. ✓ The answer that caches the full document instead of truncating it.
• •Expecting batching to reduce latency. ✗ An answer claiming the Message Batches API speeds up response time. ✓ The answer stating batching reduces cost (~50%) at the cost of being slower by design, fitting latency-tolerant jobs only.
• •Confusing cost levers with latency levers generally. ✗ An answer applying a cost-only lever (batching) to a latency-sensitive problem. ✓ The answer matching caching/smaller-model to latency-sensitive problems and batching to cost-only, latency-tolerant ones.