Accuracy-Latency-Cost Tradeoffs at Scale
AdvancedDesign observability and justify accuracy-latency tradeoffs at scale · Difficulty 4/5
Explanation
Every integration decision that touches accuracy also touches latency and cost. The architect's job is not to eliminate the tradeoff but to make it explicit and justify it against a stated requirement.
The Tradeoff Menu
- Adding a reranking step improves accuracy but adds latency and cost.
- Retrieving more chunks raises recall but bloats context and slows generation.
- **Prompt Caching a large repeated context** cuts latency and cost with *no* accuracy loss — this is the rare free win in the menu.
- A smaller model or fewer retrieved chunks cuts latency/cost at some accuracy risk.
Stating the Dominant Constraint
The architect states which constraint dominates — accuracy, latency, or cost — and justifies the chosen configuration against it. A support-triage integration with a strict SLA might accept a small accuracy hit from fewer retrieved chunks; a compliance-review integration might accept added latency from reranking because accuracy is non-negotiable there.
Common exam traps
- Optimizing latency the SLA doesn't require by dropping retrieval quality the task does need — or the reverse, over-investing in accuracy the task doesn't require at the cost of latency the SLA does need. Tie every knob to the stated accuracy/latency/cost requirement, not to a default preference for speed or precision.
Key Takeaways
- Reranking improves accuracy but adds latency and cost
- Retrieving more chunks raises recall but bloats context and slows generation
- Prompt caching a large repeated context cuts latency and cost with no accuracy loss
- A smaller model or fewer retrieved chunks cuts latency/cost at some accuracy risk
- Justify every knob against the stated accuracy/latency/cost requirement, not an assumed default
Glossary Terms
The practice of directing requests to different Claude model tiers based on assessed complexity and requirements. A common pattern uses a fast, cheap model (Haiku) to classify task complexity, then routes to Sonnet or Opus accordingly.
A Claude API feature that caches frequently-used prompt content (system prompts, large documents, tool definitions) to reduce cost and latency on repeated API calls. Cached tokens are billed at a discounted rate. Cache has a TTL that resets on each use. Must be enabled by marking content with cache_control.
A pattern that dynamically retrieves relevant information from an external knowledge base and injects it into the context window based on the current query. Allows Claude to reason over large document sets without fitting everything in context at once.
The recurring architectural tension where an integration decision that improves accuracy (e.g., reranking, retrieving more chunks) typically adds latency and cost, and vice versa. The architect's job is not to eliminate the tradeoff but to make it explicit and justify the chosen configuration against whichever constraint the stated requirement names as dominant. Prompt caching a stable repeated context is the rare exception that improves cost and latency with no accuracy loss.
Related Concepts