Model Selection and Optimization

Knowledge of

• Tokens as the unit of text (roughly 3-4 characters of English) that limits and billing are measured in, not characters or words
• Autoregressive next-token generation: the model repeatedly predicts the next token given everything so far, appends it, and continues until a stop condition
• The context window as the maximum number of tokens (input + output) the model can consider at once
• The context window as a shared budget across the system prompt, conversation history, tool definitions, tool results, and the response
• The consequence of exceeding the context window: the request fails or content must be truncated/summarized

Skills in

• Estimating token counts for text rather than reasoning in characters or words
• Explaining model output as a sequence of next-token predictions rather than a single lookup
• Reasoning about context-window limits as a shared budget across every component of a request, not a per-component allowance
• Anticipating request failure or the need for truncation/summarization when a request would exceed the context window

Knowledge of

• Sampling from a probability distribution as the mechanism of generation, with temperature controlling how focused vs. diverse that sampling is
• That LLMs remain non-deterministic even at temperature 0 -- lower temperature reduces randomness but does not guarantee identical output
• Extended thinking as an explicit reasoning budget spent before answering, and adaptive thinking/effort levels as the model varying reasoning depth by difficulty
• That thinking modes trade billed output tokens and latency for better quality on hard tasks -- they are not free
• Zero-shot, one-shot/single-shot, and multi-shot/few-shot as the fundamental spectrum of example-based prompting, and what adding examples costs

Skills in

• Choosing temperature to reduce output variance while recognizing exact reproducibility is never guaranteed
• Testing LLM-backed systems with evals rather than equality assertions, because of inherent non-determinism
• Deciding when a task's difficulty justifies the added tokens/latency of extended or adaptive thinking
• Selecting zero-shot, one-shot, or few-shot prompting based on how much the task benefits from examples versus the added input-token cost

Knowledge of

• The official Python and TypeScript SDKs as a convenience layer over the REST API handling auth, serialization, typed errors, retries, and streaming
• That the SDK can always be bypassed in favor of raw HTTPS + JSON calls to the same underlying API
• Sync vs. async clients, and async as the way to parallelize independent, I/O-bound calls
• Streaming via persistent event streams (server-sent events) as the mechanism for incremental token delivery
• That streaming/websocket transport is a delivery-mechanism concern, not a token-cost concern

Skills in

• Choosing between the SDK and raw REST calls based on whether the convenience layer's handling (retries, typed errors, streaming helpers) is needed
• Using an async client to parallelize independent, I/O-bound requests rather than serializing them
• Recognizing streaming/websocket choices as changing how tokens are delivered, not how many tokens are billed

Knowledge of

• The three model tiers -- Haiku (fastest/cheapest/lightest), Sonnet (balanced workhorse), Opus (most capable, highest cost/latency) -- and their typical task fit
• Right-sizing: matching model capability to task difficulty and value rather than defaulting to the most capable tier
• The quality vs. latency vs. cost tradeoff triangle as the frame for model choice
• Mixed-model architectures that route cheap steps to Haiku and hard steps to Opus/Sonnet within one workflow
• The risk of breaking behavior changes across model releases, and re-running evals with a pinned version before upgrading

Skills in

• Matching model tier to task difficulty and value instead of defaulting to the most capable model available
• Designing mixed-model workflows that assign each step to the tier fitting its difficulty/cost/latency profile
• Factoring adaptive-thinking support into tier selection when a workflow step needs that capability
• Re-running evaluations against a pinned model version before adopting a new release

Knowledge of

• That input and output tokens are priced differently, with output tokens typically more expensive than input tokens
• Separate cache write/read rates as part of the overall per-token pricing structure
• Cost modeling as estimating tokens times per-token price across expected traffic
• The usage field on every response (input/output/cache tokens) as the mechanism for tracking actual cost and detecting bloat
• Capping max_tokens to a realistic output length as a lever to avoid paying for runaway generations

Skills in

• Modeling expected cost by estimating token volume per request type times the applicable per-token rate
• Instrumenting the usage field on responses to monitor real spend and catch unexpectedly large requests
• Setting max_tokens to a realistic ceiling rather than leaving it unbounded

Knowledge of

• Prompt caching: caching a stable, reused prefix (system prompt, long documents, tool definitions) with cache_control so it's discounted on reuse
• That cache reads are heavily discounted versus normal input tokens, while a cache write costs slightly more than a normal input token
• Cache checkpointing as caching at multiple points in a growing, incrementally-extended prompt so it stays mostly cached
• The Message Batches API's roughly 50% per-token discount for latency-tolerant bulk jobs processed within 24 hours
• That batching reduces cost, not latency -- it is slower by design, unlike caching or right-sizing the model, which reduce latency

Skills in

• Applying cache_control to a large, stable, reused prefix rather than truncating it to save tokens
• Using cache checkpointing to keep a long, growing prompt mostly cached as it's incrementally extended
• Routing latency-tolerant, high-volume jobs to the Message Batches API for the per-token discount
• Distinguishing cost levers (batching, caching) from latency levers (caching, smaller models, trimmed context) when optimizing a workflow