Domain 5: Model Selection and Optimization
16.8% of examExplain tokens, context windows, and autoregressive generation
Key Points
- A token is roughly 3-4 characters of English; limits and billing are measured in tokens, not characters or words.
- Generation is autoregressive: the model predicts one next token from everything so far, appends it, and repeats until a stop condition.
- The context window is the max tokens (input + output combined) a model can consider in one request; exceeding it fails the request or forces truncation/summarization.
- The window is one shared budget across system prompt, conversation history, tool definitions, tool results, and the response -- not separate per-category allowances.
Decision Rules
When: Estimating whether a request will fit in the context window
→Sum tokens across system prompt + history + tool defs + tool results + expected response, not just the user's message.
When: Reasoning about a bigger context window
→Recognize it permits more tokens, but does not make tokens cheaper or curation unnecessary.
✗ Anti-Patterns to Reject
- Assuming token counts map cleanly onto word or character counts.
- Assuming tool results have a separate budget from conversation history rather than one pooled budget.
Reason about sampling, non-determinism, thinking modes, and prompting fundamentals
Key Points
- Generation samples from a probability distribution at each step; temperature controls how focused (low) vs. diverse (high) that sampling is.
- Temperature 0 reduces randomness but does NOT guarantee identical output -- LLMs remain non-deterministic.
- Extended thinking is an explicit reasoning budget spent before the answer; adaptive thinking/effort levels vary depth automatically by difficulty -- both cost billed output tokens plus latency.
- Zero-shot, one-shot, and multi-shot/few-shot are the fundamental spectrum of example-based prompting; more examples cost more input tokens.
- Because exact reproducibility is never guaranteed, test with evals, not equality assertions.
Decision Rules
When: Asked whether temperature 0 makes Claude deterministic
→Answer no -- it reduces randomness but does not guarantee identical output.
When: A task's difficulty genuinely benefits from deeper reasoning
→Apply extended or adaptive thinking, accepting the added tokens and latency.
When: Testing an LLM-backed system
→Use evals with success criteria, not exact-match equality asserts.
✗ Anti-Patterns to Reject
- "Temperature 0 makes Claude deterministic."
- "Extended thinking is free and always on."
- Writing tests that assert exact string equality against LLM output.
Distinguish SDKs from raw REST and sync/async/streaming transport
Key Points
- The official Python and TypeScript SDKs are a convenience layer over the REST API (auth, serialization, typed errors, retries, streaming helpers), not a separate protocol.
- Raw HTTPS + JSON calls to the same API are always available as a fallback to the SDK.
- Async clients parallelize independent, I/O-bound calls; sync clients block per call -- this affects wall-clock time, not token cost.
- Streaming (server-sent events) and websockets (bidirectional) are transport/delivery concerns -- they change when/how tokens arrive, not how many are billed.
Decision Rules
When: Several unrelated model calls can run concurrently
→Use the async client rather than awaiting each one sequentially.
When: Asked whether streaming reduces cost
→Answer no -- it only changes delivery timing, not total tokens billed.
When: A realtime/bidirectional integration (e.g., voice) is needed
→Consider websockets rather than one-directional server-sent events.
✗ Anti-Patterns to Reject
- Treating the SDK as a fundamentally different API from REST.
- Assuming a streaming or websocket choice changes token cost or count.
Select among Claude model tiers against quality/latency/cost tradeoffs
Key Points
- Three tiers: Haiku (fastest/cheapest, high-volume/simple tasks), Sonnet (balanced workhorse for most production logic), Opus (most capable, highest cost/latency, hardest reasoning).
- Right-sizing means matching capability to task difficulty and value, not defaulting to the most capable tier "to be safe."
- Quality, latency, and cost trade against each other -- anchor the choice on the specific requirement.
- Mixed-model architectures route cheap steps to Haiku and hard steps to Sonnet/Opus within one workflow.
- A newer model release can change behavior even with an unchanged API contract -- pin the version and re-run evals before upgrading.
Decision Rules
When: A high-volume classification or routing step must be cheap and fast
→Use Haiku, not Opus.
When: A step is the hardest reasoning or highest-value step in a workflow
→Use Opus (or Sonnet if the value doesn't justify Opus's cost).
When: Upgrading production to a new model release
→Re-run evals and pin the version rather than switching immediately.
✗ Anti-Patterns to Reject
- "Always pick the most capable model to be safe" -- wastes cost and latency.
- Assuming a workflow needs one model tier throughout rather than a per-step mixed-model design.
Model token cost and track usage
Key Points
- Input and output tokens are priced differently; output tokens are typically more expensive than input tokens.
- Cache writes and cache reads have their own distinct rates within the overall pricing structure.
- Cost modeling = estimated tokens per request type x per-token price, summed across expected traffic.
- The usage field (input/output/cache tokens) on every response is the instrumentation for tracking real cost and catching bloat.
- Capping max_tokens to a realistic ceiling prevents paying for runaway generations.
Decision Rules
When: Modeling expected cost at scale
→Estimate token volume per request type times the applicable per-token price, summed across expected traffic.
When: Monitoring real spend
→Instrument the usage field on every response rather than relying only on modeled estimates.
When: Setting max_tokens
→Cap it to a realistic output length rather than leaving it unbounded.
✗ Anti-Patterns to Reject
- Assuming input and output tokens are priced the same.
- Treating cost modeling as a one-time estimate rather than something the usage field lets you continuously verify.
Reduce cost and latency with prompt caching and the Message Batches API
Key Points
- cache_control caches a stable, reused prefix; cache reads are heavily discounted, cache writes cost slightly more than normal input -- the savings come from reuse, not the first call.
- Cache checkpointing keeps a long, incrementally-growing prompt mostly cached as new content is appended.
- The Message Batches API gives a roughly 50% per-token discount for asynchronous jobs completed within 24 hours.
- Batching reduces cost, not latency -- it is slower by design; caching and right-sized models reduce both cost and latency.
Decision Rules
When: Many requests share a long, unchanging prefix
→Cache it with cache_control rather than resending or truncating it.
When: A workload is latency-tolerant, high-volume, and cost-sensitive
→Route it to the Message Batches API.
When: Optimizing a latency-sensitive step
→Use caching or a smaller model, not batching -- batching does not improve latency.
✗ Anti-Patterns to Reject
- Assuming batching improves latency because it processes "in bulk."
- Truncating a needed stable document to save tokens instead of caching the full content.