5.3 Risks, Failure Modes & Human-in-the-Loop Validation

"The model might be wrong" is true, and also nearly useless as an engineering statement. It tells you nothing about WHAT kind of wrong, and different kinds of wrong need completely different fixes. If you ground the model's answers in retrieved documents to fix hallucination, that does absolutely nothing for a PHI leak in the output. If you pin the model version to fix drift, that does nothing for a human blindly trusting a confident-sounding answer. An architect's actual job in Task Statement 5.3 is to name the failure mode PRECISELY enough that the mitigation follows from the name.

There are six named failure modes worth having cold. Two of them — prompt injection and jailbreaks — you already met in Lesson 5.2, because they belong to this same taxonomy; the other four are new. Think of this section as a diagnostic checklist: when something goes wrong in production, walking through these six names should tell you, within a sentence or two, which family of mitigation to reach for.

Start with the two failure modes that live entirely inside the model's own output, before any tool or system gets involved. HALLUCINATION is confident but fabricated output — the model states something false with exactly the same fluent, assured tone it uses for something true. This is the trap at the heart of Task Statement 5.3: nothing about HOW an answer is phrased tells you whether it's correct. Confidence is not correctness, full stop, and a high-stakes claim needs to be grounded and verified regardless of how certain the model sounds saying it.

The mitigations follow directly from what hallucination actually is — a claim untethered from a real source. RETRIEVAL GROUNDING anchors claims to documents you retrieved and can point to, rather than the model's parametric memory alone. CITATIONS force the model to attribute a claim to a specific source, which both helps a reviewer check it and discourages claims with no source to cite. CONSTRAINED CLAIMS scope what the model is allowed to assert — "answer only using the provided context" — so it can't wander into unsupported territory. And an explicit "SAY I DON'T KNOW" allowance removes the pressure that produces a lot of hallucination in the first place: a model that's implicitly expected to always produce SOME answer will sometimes fabricate one rather than admit it doesn't know. None of these are increasing temperature for variety, trusting the model's self-reported confidence, or removing the eval set — those don't touch the actual mechanism.

NON-DETERMINISM is a quieter failure mode, but no less real: the same input can yield different output from one call to the next, or across model versions. This isn't a bug to be stamped out — it's a property of how these models generate text — so the right response is to design AROUND it rather than assume reproducibility. That means evaluation suites that sample multiple runs rather than trusting one lucky pass, and human review specifically on high-stakes paths where a single unlucky output could matter.

The next two failure modes have nothing to do with whether any individual sentence is true — they're about what surrounds the model. DATA LEAKAGE is the model surfacing PII or secrets it shouldn't have exposed: a customer-service agent that has access to every customer's record accidentally reveals one customer's details to another, or an internal tool's error message echoes a database credential back into a user-facing response. The fix is exactly the permissions and output layers from Lesson 5.1 — scope what data the agent can access in the first place, and validate output before it leaves so a leak is caught even if it occurs.

MODEL MISMATCH or DRIFT is what happens when the ground shifts under you: the same prompt, sent to a newer model version, produces meaningfully different behavior — a different tone, a different willingness to use a tool, a different edge-case decision. This isn't hypothetical carelessness; it's a predictable consequence of upgrading a component you don't fully control the internals of. The fix is to PIN model versions in production rather than floating to "whatever's latest," and to RE-RUN your evaluation suite (the Domain 4 machinery) before deliberately upgrading, so a behavior change is caught in testing rather than discovered by a customer.

The last named failure mode is the only one that isn't really about the model at all — it's about US. OVER-RELIANCE, or AUTOMATION BIAS, is the human tendency to trust confident-sounding AI output uncritically, simply because it's fluent and arrives quickly. And here's why it deserves its own line item rather than folding into hallucination: even a system that hallucinates relatively rarely can still fail badly in practice if the humans downstream have stopped checking its work.

Think about what happens over the first few weeks a reviewer uses a new AI-assisted tool. Early on, they check everything carefully. The tool is right, again and again. Over time, checking starts to feel like a formality — the tool has earned trust, and trust quietly turns into a rubber stamp. That's automation bias setting in, and it's a genuinely dangerous pairing with hallucination: a system that produces occasional confidently-wrong output, reviewed by humans who've stopped expecting to find anything wrong, is worse than either problem alone. The mitigations are TRANSPARENCY — making it clear what output is AI-generated and what it's grounded in, so a reviewer has a concrete reason to check rather than a vague sense they should — and mandatory HUMAN VALIDATION specifically on high-stakes output, designed as an actual review step rather than a review step that exists on paper but has quietly become optional in practice.

This is exactly why hallucination mitigation and automation-bias mitigation have to be designed TOGETHER. Grounding reduces how often the model is wrong. Transparency and mandatory validation ensure humans still catch it on the occasions it is. Neither substitutes for the other — a well-grounded system with no real review step still fails when grounding isn't enough, and a heavily-reviewed but ungrounded system just wastes reviewer time on errors that better grounding would have prevented in the first place.

Automation bias tells you humans need to stay engaged; it doesn't tell you WHERE. Task Statement 5.4 answers that with a specific framework, and it's worth learning as a framework rather than a vague instinct that "AI needs supervision." The deciding criteria are STAKES and REVERSIBILITY, evaluated together, not a general sense of how important a task feels.

High-stakes, irreversible actions — a financial transaction, an account deletion, a medical or legal output someone will act on — require human approval BEFORE execution. The reasoning is blunt: if it goes wrong, you cannot undo it, so the check has to happen while there's still something to check. Lower-stakes or genuinely reversible actions don't need that same pre-execution gate on every single instance, but they still benefit from DETECTIVE REVIEW — sampling and auditing a subset of outputs even though no individual one is blocked pending approval. And running underneath both is the ESCALATION PATH: the system itself should recognize when its own confidence is low, or when a request falls outside its policy, and defer to a human rather than guessing and proceeding anyway.

Here's the worked example that ties this back to Lesson 5.1's deterministic/probabilistic line: a workflow that issues refunds automatically, where refunds are high-value and hard to reverse, needs human-in-the-loop approval before execution — not trusting the model's judgment to keep things fast, and not merely logging the refunds and reviewing later. That second option is detective, not preventive, and for a hard-to-reverse action, detective isn't enough (this is the same point from 5.1.4, now applied specifically to the human-review design problem). The approval gate itself then has to be DETERMINISTIC — paired with a hook or permission rule so the model literally cannot execute the refund without the human step completing. A human-in-the-loop requirement that the model could simply talk its way around isn't really a control at all.

The stakes-and-reversibility framework you just learned tells you WHERE to place a human approval gate. It says nothing, by itself, about a failure mode that shows up specifically when that framework is applied too liberally — when "when in doubt, route it to a human" quietly becomes the default answer for every borderline case, not just the genuinely high-stakes ones. The named failure mode for this over-application is CONSENT FATIGUE, and it's one of the more counter-intuitive risks in this domain, because on the surface it looks like an excess of caution rather than a shortfall of it.

Consent fatigue is what happens when routing too many decisions to human review erodes the quality of that review over time. A human-in-the-loop control only delivers the safety benefit it's designed for if the human on the other end is actually evaluating each item — reading it, weighing it, deciding. Push the volume of items routed to that reviewer high enough, while keeping the context given for each item thin enough, and the reviewer's actual behavior shifts from evaluation to RUBBER-STAMPING: clicking approve because the queue is long, because the last fifty items were all fine, and because there's no real signal in front of them explaining WHY this particular item was even flagged. The human-in-the-loop control still sits on the architecture diagram at that point — but it has stopped doing the job it was drawn there to do.

This is the predictable consequence of over-applying the framework you just learned, not a separate problem to learn from scratch. The framework says: route genuinely high-stakes, low-confidence, or out-of-policy items to a human, and let lower-stakes, reversible, high-volume actions rely on detective sampling instead. Consent fatigue is what happens when that discrimination step gets skipped — when an architect, or an over-cautious default configuration, routes EVERYTHING to human review regardless of actual stakes: every borderline case, every slightly-unusual request, every action the system isn't 100% confident about, no matter how low the real stakes or how reversible the real action. The review queue balloons, and a reviewer facing a high-volume queue of low-context items does exactly what any reasonable human would do under those conditions: they start approving quickly instead of evaluating carefully.

Picture the reviewer's actual screen: a queue of items, each with an approve/reject button, and little else — no visibility into why THIS item was flagged. Was it a low confidence score? An unusual pattern? A specific policy trigger? Just the item and a binary choice, fifty times an hour. That's not a character flaw in the reviewer; it's the predictable outcome of handing anyone an unbounded stream of low-context, low-explanation decisions to make quickly. The design failure sits upstream of the reviewer, in how the queue was built — which is exactly why the fix has to be architectural, not a pep talk about paying closer attention.

The fix is not to abandon human-in-the-loop review — it's to apply the same stakes/reversibility/confidence criteria already established for HITL placement, and to design the review EXPERIENCE so a genuine evaluation is actually possible.

• 1.ROUTE SELECTIVELY, per the stakes/reversibility/confidence criteria already in the framework -- not everything borderline, only what the framework actually calls for (high-stakes, irreversible, low-confidence, or out-of-policy). Lower-stakes volume goes to detective sampling instead, exactly as the framework already prescribes.
• 2.GIVE REVIEWERS THE FLAGGING REASON and relevant context -- not just raw input/output. A reviewer who sees "flagged: confidence 0.42, below the 0.7 threshold for this action type" or "flagged: requested refund exceeds the $500 auto-approval limit" can actually evaluate the specific concern. A reviewer handed only the raw request and a button cannot.
• 3.PERIODICALLY AUDIT REVIEW-QUEUE APPROVAL RATES. A queue that has silently drifted to 99%+ approval is a strong, measurable signal that the queue has grown consent-fatigued and the review step is no longer adding real signal -- it has become theater. This doesn't require guessing at reviewer state of mind; it's a number pulled directly from the system's own logs.

Worked example: an architect, having correctly learned that high-stakes actions need human approval, over-applies the lesson by routing every tool call below a 90% confidence score to a human reviewer, regardless of what the tool actually does. Within a few weeks the reviewer is approving 400 items a day, most of them low-stakes, easily-reversible actions (rephrasing a canned reply, re-ordering a list) that happened to score below the arbitrary cutoff for unrelated reasons. Overwhelmed, the reviewer starts approving almost everything without close reading -- and a genuinely risky item that also happens to score below 90% now sails through the same rubber-stamped queue as everything else. The fix isn't removing the human reviewer; it's routing only the genuinely high-stakes, low-confidence, or policy-relevant subset to that reviewer, with the specific reason attached, and letting the low-stakes majority go through detective sampling instead.

Two task statements, one family of traps: treating a specific failure or risk generically instead of naming it precisely, and treating human review as a blanket policy instead of a control fitted to stakes and reversibility.

• •Vague risk naming. ✗ "The model might be wrong, so add more review everywhere." ✓ Name the specific failure mode (hallucination? leakage? drift?) and apply its specific mitigation.
• •Confidence as correctness. ✗ Trusting a fluently-worded, confident answer on a high-stakes question. ✓ Ground and verify; confidence is not correctness.
• •Ignoring automation bias. ✗ Assuming a human reviewer will keep catching errors indefinitely just because a review step exists on paper. ✓ Design transparency in so reviewers have a reason to keep checking.
• •Applying HITL uniformly. ✗ "Add human review to everything" or, at the other extreme, "the model's confident, skip review." ✓ Decide by stakes and reversibility — preventive gates for high-stakes/irreversible, detective sampling for the rest.
• •Trusting the model's judgment on a hard-to-reverse action. ✗ "The model can decide when to escalate a refund." ✓ The approval gate must be deterministically enforced, not left to the model's own discretion.

You now have the six named failure modes and their distinct mitigations, the confidence-is-not-correctness principle, the automation-bias trap, and the stakes-and-reversibility framework for human-in-the-loop design. The exercise asks you to apply both halves to one system.

You've now covered the risks a Claude system runs into on its own (5.3) and where to put humans in the loop to catch them (5.4). The final lesson turns outward — to the regulatory and ethical obligations that apply once this system is actually handling real people's data and decisions, closing out Domain 5.