6.3 Feedback Loops, Expectation Alignment & SLAs

Imagine a contractor tells you your kitchen renovation will be done in two weeks, knowing full well it always takes six. You'd rather they'd said six weeks up front — at least then you could plan around it. The two-week promise doesn't make the renovation faster; it just guarantees a moment, six weeks from now, when trust breaks. An SLA works exactly the same way. It's a promise, and a promise the architecture can't keep is worse than no promise at all, because it converts an engineering limitation into a broken commitment.

So the discipline behind Task Statement 6.3 starts here: SLAs must be grounded in what the chosen architecture can ACTUALLY deliver, not in what would be convenient to promise. Latency and reliability targets have to match the model tier you picked, the number of retrieval or reranking hops in the pipeline, and the infrastructure underneath it — the same plan and feature tradeoffs you made (or will make) elsewhere in the design. An SLA set before those choices are locked in, or set independently of them, isn't a commitment. It's a guess wearing a commitment's clothes — the same failure mode Lesson 6.1 warned about for architecture proposed before discovery finished, just one lifecycle stage later.

There's one scenario the exam returns to so often it's worth memorizing by shape rather than by exact wording: committing to sub-second latency on a multi-step agentic pipeline that includes reranking. Walk through why this specific combination is a trap, because the reasoning generalizes to every SLA you'll ever set.

A multi-step agentic pipeline, by definition, means the model reasons, calls a tool, waits for the result, and reasons again — potentially several times — before it has an answer (exactly the loop mechanics from Domain 1). Each of those round trips costs real time. Reranking adds another cost on top: after initial retrieval, a separate scoring pass re-orders candidates for relevance, which is additional latency in exchange for additional accuracy — the same "we gain X, we pay Y" tradeoff from Lesson 6.2, just paid in milliseconds instead of dollars. Stack enough of these steps together and a sub-second total response time simply isn't physically available from the components you chose. Promising it anyway doesn't change the physics; it just schedules the moment you'll have to explain why the promise broke.

The fix is always the same shape, regardless of the specific numbers involved: measure — or, before you have a working system, estimate from the latency of the individual components — what the architecture can actually deliver, and commit to THAT. If the required SLA is genuinely non-negotiable (say, a live chat interface truly needs sub-second responses), the fix isn't to promise the impossible number; it's to change the architecture first — fewer steps, a faster model tier, cached or precomputed results — until the design can honestly hit the number you're being asked to promise.

There's a second, quieter way to overcommit in an SLA, and it has nothing to do with latency: implying the system will always be right. LLM-based systems are probabilistic by nature — the same input can, in principle, produce a slightly different output on a different run, and even a well-tuned system will occasionally be wrong on genuinely hard or ambiguous inputs. That's not a defect to be engineered away entirely; it's a structural property of the technology you're deploying.

So part of setting a realistic SLA is being honest that quality is a BOUND, not a guarantee. The commitment you can honestly make isn't "the system will always be right" — it's "the system is evaluated against a defined quality bar and continuously monitored to stay within it." That's a real, defensible promise: it tells the stakeholder exactly what's being measured and what happens if the measurement slips. "Always right" is not a promise at all — it's a claim that will eventually be falsified by a single wrong answer, at which point the stakeholder doesn't just lose confidence in that one answer, they lose confidence in every commitment you made alongside it.

Everything so far in this lesson has been about setting the SLA correctly at one point in time. But expectation management doesn't end at sign-off — it's continuous, for a simple reason: nothing about the system holds still. Model providers ship new versions. The data flowing through the system drifts as the business changes. Traffic grows past what the original design was sized for. Any one of these can shift what's actually feasible, in either direction — sometimes for the better.

Two habits keep expectations matched to reality across that changing landscape. First, establish an actual FEEDBACK LOOP — a defined mechanism for how stakeholders report issues, and a defined pipeline for how those reports feed prioritized iteration. Without that mechanism, issues surface informally and inconsistently — a complaint in a hallway conversation, a frustrated email — and prioritization becomes reactive guesswork instead of a deliberate process. Second, RE-ALIGN PROACTIVELY: when a model version changes, a data shift occurs, or a scaling need arrives, tell the stakeholder what that means for their SLA before they discover the gap on their own. Being the one who raises the issue, rather than the one who gets caught by it, is the entire difference between managing expectations and merely reacting to their collapse.

• 1.Establish a defined mechanism for how stakeholders report issues — not an informal, ad hoc channel.
• 2.Feed those reports into a prioritized iteration process, so the loop actually changes something.
• 3.Watch for the events that shift feasibility: a new model version, a data change, a scaling need.
• 4.When one of those events occurs, communicate proactively — before the stakeholder discovers the gap themselves.

A realistic SLA tells a stakeholder what latency and quality to expect. It's a separate, equally important commitment to tell them what the BILL will look like -- and this is a place where architects who are otherwise careful about expectation-setting still routinely drop the ball, because the cost conversation that happened during discovery and design was almost always priced against pilot-scale numbers, not production-scale ones.

The core guidance: architects should proactively give stakeholders a CONSUMPTION FORECAST at expected PRODUCTION volume -- not pilot or proof-of-concept volume -- before the first invoice arrives. "Proactively" and "before the first invoice" are both load-bearing words here. This has to happen before the bill lands in someone's inbox, not as a reactive explanation after a stakeholder is already alarmed by a number nobody warned them about.

Why is pilot-scale cost specifically a trap? Because a pilot's traffic footprint is a rounding error next to what the live system faces once real users show up in real volume -- a jump measured in tens or hundreds of times over, not a modest bump. A pilot that ran 50 requests a day and cost $30 a month is not a system that will cost $3,000 a month at 5,000 requests a day just by linear scaling -- it might cost far more, or in some cases proportionally less, depending on how costs actually scale with volume in the specific architecture. The number that matters is the number at the volume the system will actually run at once it's live, not the number a small pilot happened to produce.

Here's the sharpest part of the lesson: the damage from an unforecasted cost shock happens REGARDLESS of whether the technical design was sound. An architecture can be exactly the right choice -- correctly scoped, well-engineered, delivering precisely the quality and latency it was supposed to -- and still produce a moment where the stakeholder's trust in the architect collapses, purely because the invoice arrived as a surprise. Cost is one of the business-value pillars, alongside latency, quality, and risk, that a stakeholder weighs a decision against (recall the audience-anchoring framework from earlier in this domain); a surprise on that pillar undoes trust just as thoroughly as a missed SLA would, even when every other dimension of the system performed exactly as promised.

Here's an original worked example. Suppose a pilot tests a customer-support drafting assistant at 50 requests a day for three weeks, and the pilot's bill comes in around $30 for the month. The stakeholder mentally files the system as "cheap to run" -- a reasonable inference from the only number they've seen. Production rollout then expands usage to 4,000 requests a day, roughly an 80x increase in volume. If cost scaled purely linearly with request count, the stakeholder might brace for something in the neighborhood of $2,400 a month.

But the actual architecture has a fixed per-session context-loading step: each new customer session requires loading and formatting a substantial chunk of that customer's account history and prior ticket context into the prompt before the first request in the session can be answered, and that loading cost is fixed per session regardless of how many requests happen within it. At pilot volume, testers ran long extended conversations, so the fixed per-session overhead was amortized across many requests and barely visible in the per-request average. At production volume, real customer sessions are short -- often one or two requests before the ticket resolves -- so the fixed overhead gets paid over and over, on nearly every request, instead of being spread thin. The per-request cost at production volume isn't the pilot number scaled up; it's a meaningfully higher per-request number, because the mix of session lengths changed the underlying economics. The result: an $800-a-month bill instead of anything close to the $2,400 the stakeholder might have braced for from linear scaling -- and nobody forecasted that figure for them in advance, so it surfaces for the first time on the actual invoice.

The fix isn't a different architecture -- the per-session context loading may well be the right design choice, delivering real quality benefits from having full account context available. The fix is that the $800 figure, or something close to it, should have been forecasted and communicated BEFORE the production rollout, using production-volume assumptions and the actual session-length distribution expected in real usage, not the pilot's atypical long-session pattern.

This is the same discipline as grounding an SLA in measured capability rather than aspiration, applied to the cost dimension instead of the latency dimension: just as an SLA has to be grounded in what the chosen model tier and pipeline steps can actually deliver, a cost forecast has to be grounded in what the chosen architecture will actually cost AT THE VOLUME IT WILL ACTUALLY RUN AT, not at whatever volume happened to be convenient during the pilot. And it connects directly to the feedback-loop discipline from the previous section: if a scaling event is coming, the right move -- as with a model version change or a data shift -- is to communicate what it means before the stakeholder discovers the gap themselves.

You now have the SLA toolkit: ground commitments in the measured architecture, watch for the multi-step-plus-reranking latency trap, be honest about non-determinism, and pair a defined feedback loop with proactive re-alignment. The exercise turns this into something concrete you can reuse.

A realistic SLA, backed by a real feedback loop, is the commitment that has to survive contact with reality — which is exactly what documentation and implementation guidance, the subject of Lesson 6.4, are meant to protect. If the rationale behind every choice is written down, the team that eventually re-aligns an SLA after a model update will know exactly which tradeoff they're allowed to revisit.