4.3 Optimizing Cost, Latency & Token Usage
4.3.1 Optimization Is a Tradeoff, Never a Free Lunch
Once a system passes its eval, the natural next question is: can we make it cheaper or faster? The answer is almost always yes — there are real, well-understood levers for reducing cost, latency, and token usage in a Claude-based system. But every one of those levers touches the same input or output that your accuracy metric depends on, which means every optimization is really a hypothesis: "this change will save money/time without costing quality." Task Statement 4.5 is about knowing the five real levers, and about treating every one of them as a hypothesis that has to be checked, not a free win you can just ship.
It helps to think about WHERE cost and latency actually come from. Cost is roughly tokens-in plus tokens-out, priced per model tier. Latency is roughly time-to-first-token plus generation time, which scales with how much the model has to read and how much it has to write. Every optimization lever below targets one of these specific drivers — which is why applying the wrong lever to the wrong driver (like expecting a batching API to speed up a single urgent request) doesn't produce the saving you expected.
The one idea to hold onto
Optimization is a tradeoff against measured quality, never applied blind. Before adopting any cost or latency change, you must be prepared to re-run the eval and confirm the quality bar still holds — a saving that quietly drops accuracy below the bar is a regression, not a win.
4.3.2 The Five Levers
There are five standard techniques for reducing cost, latency, and token usage. Learn them as a set, because exam scenarios often ask you to pick the single correct lever for a specific situation — and the wrong-answer options are almost always one of the OTHER four levers applied to a situation it doesn't fit.
| Lever | What it does | Targets |
|---|---|---|
| Prompt caching | Cut latency and cost on repeated stable prefixes with no accuracy loss — order stable content first so the cacheable portion is contiguous | Repeated input tokens |
| Right-size the model | Route each step to the cheapest tier that still passes the eval; reserve Opus / extended thinking for steps that actually need that capability | Per-token pricing |
| Trim the context | Prune stale tool output and over-fetched chunks; fewer tokens means lower cost AND less context rot | Total input volume |
| Batch | Batch latency-tolerant bulk jobs via the async Batches API to cut cost — it does NOT reduce per-request latency | Throughput economics |
| Cap max_tokens | Bound output cost with an appropriate ceiling | Output volume |
The five optimization levers, each targeting a different cost or latency driver. Applying a lever to the wrong driver — like batching for a single urgent request — doesn't save what you expected.
Two of these deserve a closer look because the exam likes to probe the exact boundary of what they do and don't accomplish. Prompt caching works by reusing a stable prefix — the same system prompt, the same long reference document, the same tool definitions — across many requests, so the model doesn't have to reprocess that prefix from scratch every time. For this to work, the stable content has to come FIRST in the prompt, before anything that varies request to request; if you interleave stable and variable content, the cache can't find a long enough unchanged prefix to reuse.
The Batches API is the one lever people most often misuse. It's asynchronous and cheaper per token — genuinely useful for large, latency-tolerant jobs like reprocessing a million documents overnight. But it does NOT reduce per-request latency; a batch job trades immediacy for cost efficiency. If a scenario describes a user waiting in real time for a single response, batching is not the fix, no matter how much cheaper it is per token — that's a throughput optimization being misapplied to a latency problem.
# Ordering for prompt caching: stable content first, variable content last
messages = [{
"role": "user",
"content": [
{
"type": "text",
"text": SYSTEM_POLICY_DOCUMENT, # long, stable, reused every request
"cache_control": {"type": "ephemeral"},
},
{"type": "text", "text": user_specific_question}, # varies every request, goes last
],
}]
# Because the stable block comes first and is marked for caching, repeated
# requests reusing this exact prefix skip reprocessing it — lower cost, lower latency.4.3.2 — Key Concept
Five levers, five different drivers: prompt caching (repeated input tokens, stable content first), model right-sizing (per-token pricing), context trimming (total input volume, also reduces context rot), batching (throughput economics for latency-tolerant bulk jobs — NOT per-request latency), and capping max_tokens (output volume).
4.3.3 Right-Sizing and Context Trimming
The remaining three levers are more about discipline than about a specific API feature. RIGHT-SIZING THE MODEL means resisting the default instinct to run every step of a pipeline on the most capable (and most expensive) model tier. Instead, route each step to the cheapest tier that still passes the eval — a classification step that a smaller, faster model handles perfectly well shouldn't be routed through your most expensive tier just because it's available. Reserve the most capable models and extended thinking for the steps that genuinely need that reasoning depth: complex synthesis, ambiguous judgment calls, multi-step planning.
TRIMMING THE CONTEXT means pruning stale tool output and over-fetched chunks before they pile up in the conversation. This lever is unusual among the five because it's a genuine two-for-one: fewer tokens directly means lower cost, AND it means less CONTEXT ROT — the degradation in output quality that can occur as a conversation accumulates irrelevant or outdated content the model still has to attend to. Unlike batching (a pure cost play) or right-sizing (a pure cost play with a quality floor), context trimming can improve BOTH cost and quality at once, which is why it's worth doing even when cost isn't the primary concern.
Capping max_tokens is the simplest lever and the easiest to misuse: set the ceiling too low and you truncate content the task genuinely needs (which shows up later as the stop_reason: max_tokens symptom from Lesson 4.2), set it comfortably higher than needed and you've simply capped nothing. The correct value is the smallest ceiling that never truncates a legitimate response for this task — which you determine empirically, against the eval, not by guessing.
4.3.3 — Key Concept
Right-size the model per step against the eval, not by defaulting to the most capable tier everywhere. Context trimming is the rare lever that improves cost AND quality simultaneously, by reducing both token volume and context rot.
4.3.4 The Rule That Governs All Five: Verify Against the Eval
Here is the single rule that ties every lever in this lesson together, and the exam's single favorite trap in this task statement: before adopting ANY cost or latency optimization, re-run the eval you built in Lesson 4.1 and confirm the quality bar is still met. A cost win that silently drops accuracy below the bar is not an optimization — it is a regression wearing a lower invoice.
This is worth dwelling on because the five levers are NOT all equally risky. Prompt caching with stable content ordered first is genuinely free — it reuses the exact same content, just cheaper to reprocess, so there's no accuracy tradeoff to verify against in principle (though it's still good practice to confirm). But truncating a policy document to save tokens, dropping a retrieval step entirely to cut latency, or setting max_tokens below the length the task actually needs — these all trade away content the system needs, even though they superficially resemble caching ("we removed content, so it's cheaper"). The resemblance is the trap: not every token-reduction technique is equally safe, and the only way to tell the difference is to measure.
| Optimization | Accuracy risk | Why |
|---|---|---|
| Prompt caching (stable content first) | None | Reuses identical content; only reprocessing cost changes |
| Right-sizing to a cheaper model | Verify per step | A less capable model may miss what the eval was checking for |
| Trimming stale/over-fetched context | Verify, usually improves quality too | Removes noise, but risks removing something needed if done carelessly |
| Truncating a policy document to save tokens | High | Removes content the system may need to answer correctly |
| Dropping retrieval entirely | High | Removes the grounding that prevents hallucination |
| Lowering max_tokens below the needed length | High | Truncates legitimate output mid-answer |
Not all cost reductions carry equal risk. Caching is close to risk-free; the bottom three trade away content the system needs and must never be adopted without re-running the eval.
4.3.4 — Exam Trap
Optimizing cost or latency without re-running the eval is the core trap. "Cheaper is always better" is never the correct answer, and neither is compensating for a resulting quality drop by adjusting an unrelated parameter (like raising temperature). The correct workflow is always: apply the change, re-run the eval, then adopt only if the quality bar still holds.
4.3.5 Put It Together: Optimize Without Breaking Quality
You now have the five levers, know which driver each one targets, understand why caching is close to risk-free while truncation and dropped retrieval are not, and know the verification rule that governs all of them. The exercise puts that discipline into practice on a concrete pipeline.
4.3.5 — Build Exercise (45 min)
(1) Take a multi-step pipeline and order its stable system content (policy text, tool definitions) before the variable per-request content, then apply prompt caching and measure the cost/latency delta. (2) Audit each pipeline step and right-size it to the cheapest model tier that still passes the eval — confirm at least one step can move to a cheaper tier without a quality drop. (3) Add a context-trimming pass that prunes stale tool output before it's resent, and re-run the eval to confirm quality holds or improves. (4) As a deliberate negative example, truncate a policy document to save tokens and re-run the eval — observe the quality regression, and note how it superficially resembles a cost win until measured. (5) Identify one latency-tolerant bulk job in the pipeline and move it to the Batches API, confirming cost drops while per-request latency for the batch items is understood to be unaffected in the interactive sense.
Where this shows up on the exam
4.5 questions present a specific cost/latency goal and ask which lever fits — or present an optimization already applied and ask what must happen before it's adopted (always: re-run the eval). Watch for batching mismatched to a single-request latency need, and for truncation/dropped-retrieval dressed up as an optimization.
Key Takeaways
- ✓Optimization is a tradeoff against measured quality, never blind — every cost/latency change is a hypothesis that must be checked against the eval.
- ✓Five levers, five different drivers: prompt caching (repeated input, stable content first), model right-sizing (per-token pricing), context trimming (input volume + context rot), batching (throughput economics, NOT per-request latency), capping max_tokens (output volume).
- ✓Prompt caching requires stable content ordered first and contiguous; interleaving stable and variable content breaks the cacheable prefix.
- ✓The Batches API cuts cost for latency-tolerant bulk jobs but does NOT reduce per-request latency — it's the most commonly misapplied lever.
- ✓Context trimming is unusual: it can improve cost AND quality simultaneously by reducing both token volume and context rot.
- ✓Not all cost reductions carry equal accuracy risk — caching is close to risk-free; truncating content, dropping retrieval, or under-sizing max_tokens all trade away needed content.
- ✓Before adopting ANY optimization, re-run the eval; a cost win that drops accuracy below the bar is a regression, not a win — "cheaper is always better" is never the correct answer.
Check Your Understanding
Test what you learned in this lesson.
Q1.A team needs to process 2 million archived support tickets overnight for a one-time analysis. Latency for any individual ticket doesn't matter, but the total cost does. Which optimization lever fits best?
Q2.An engineer proposes reducing a chatbot's cost by truncating the reference policy document included in every prompt from 4,000 to 800 tokens, calling it equivalent to enabling prompt caching. Why is this comparison flawed?
Q3.After right-sizing three pipeline steps to a cheaper model tier to cut costs, what must the team do before shipping the change to production?
Q4.For prompt caching to actually reduce cost and latency on repeated requests, what must be true about how the prompt is structured?
Practice This Lesson