Setting Realistic SLAs Grounded in Measured Capability
AdvancedManage feedback loops, expectation alignment, and SLAs · Difficulty 3/5
Explanation
SLAs are a promise, and a promise the architecture can't keep is worse than no promise at all. Setting a realistic SLA means grounding the commitment in what the chosen design can actually deliver.
Ground Commitments in the Architecture, Not Aspiration
Latency and reliability targets must match the chosen model tier, retrieval steps, and infrastructure -- plan and feature tradeoffs (e.g., which plan tier, which model, how many retrieval/reranking hops) directly inform what's achievable. An SLA set before those choices are made, or set independently of them, is a guess.
Honesty About Non-Determinism
LLM systems are probabilistic. Part of setting a realistic SLA is being honest that quality is a bound, not a guarantee -- the system is evaluated and monitored against a quality bar, not proven perfect. Framing an SLA as "the system will always be right" sets an expectation the architecture cannot honor.
Common exam traps
- Committing to an SLA the architecture can't meet -- the canonical example is promising sub-second latency on a multi-step agentic pipeline with reranking. Multi-step agentic designs and reranking both add latency; a sub-second commitment on top of them is not grounded in measured capability.
The fix is always the same: measure (or estimate from the chosen components) what the architecture can deliver, and commit to that -- adjusting the architecture first if the required SLA is non-negotiable, rather than committing to a number the design can't hit.
Key Takeaways
- SLAs must be grounded in what the chosen model tier, retrieval steps, and infrastructure can measurably deliver -- not aspiration
- Plan/feature tradeoffs (model tier, retrieval hops, reranking) directly determine what latency/reliability targets are achievable
- Be honest that LLM output is probabilistic -- frame quality as evaluated and monitored against a bar, not guaranteed perfect
- Classic trap: promising sub-second latency on a multi-step agentic pipeline with reranking -- both add latency the SLA must account for
Glossary Terms
An asynchronous Claude API for processing multiple requests in a batch with 50% cost savings versus synchronous requests. Processing takes up to 24 hours with no guaranteed latency SLA. Does not support iterative tool use, streaming, or prompt caching. Best for scheduled, non-blocking analysis.
A commitment about a system's latency and reliability that must be grounded in what the chosen model tier, retrieval steps, and infrastructure can actually deliver, not in aspiration. A common exam trap is promising sub-second latency on a multi-step agentic pipeline with reranking -- both add latency the SLA must account for. Re-alignment is continuous: a model version, data change, or scaling need can shift what's feasible.
Related Concepts