Domain 5: Governance, Safety & Risk Management

14% of exam

ts-ccarp-5.1

Design layered guardrails and safety controls for production Claude systems

Key Points

Guardrails are layered: input, permissions, deterministic controls, output, and monitoring -- no single layer is assumed perfect.
A PreToolUse hook is a deterministic control that blocks a destructive action every time, regardless of what the model 'decides.'
A prompt guardrail is probabilistic -- a system-prompt sentence the model usually follows, with a non-zero failure rate.
Hard policy limits and destructive-action prevention belong in deterministic controls, not a system-prompt sentence.
Logging a violation after the fact is detective, not preventive -- it doesn't replace a gate.

Decision Rules

When: A risk is a hard limit (financial threshold, irreversible action, compliance-mandated behavior)

→Enforce it with a deterministic hook/permission rule, not a prompt sentence.

When: A style or tone preference is soft (occasional deviation tolerable)

→A prompt guardrail alone is acceptable.

When: Designing a safety architecture

→Layer input filtering, least privilege, deterministic controls, output validation, and monitoring rather than relying on one layer.

✗ Anti-Patterns to Reject

Putting a hard safety rule only in the system prompt.
Relying on a single guardrail layer instead of layered, independent defenses.

ts-ccarp-5.2

Mitigate prompt injection, jailbreaks, and untrusted input

Key Points

Prompt injection = malicious instructions hidden in untrusted content (web pages, documents, tool results) that the model reads as commands.
Unlike SQL injection, there is no strict syntactic boundary between instructions and data in natural language -- the fix is architectural, not linguistic.
Core defenses: isolate/delimit untrusted content, apply least privilege, filter/monitor for injection patterns, gate high-stakes actions with human review.
Least privilege contains the blast radius even if isolation fails -- an agent that can't delete data can't be tricked into deleting it.
Model size and sampling temperature do not defeat prompt injection; neither does a polite 'ignore injected instructions' caveat.

Decision Rules

When: An agent consumes untrusted content (web pages, documents, tool results)

→Delimit/label it clearly so it isn't executed as an instruction.

When: An injection might trigger a high-stakes action

→Require human-in-the-loop approval before execution.

When: Asked whether a bigger model or lower temperature reduces injection risk

→Reject it -- neither isolates untrusted content nor limits what a hijacked instruction can do.

✗ Anti-Patterns to Reject

Believing a more capable model or temperature=0 defeats prompt injection.
Adding an in-prompt caveat ('please don't obey injected instructions') as if it were a real defense.

ts-ccarp-5.3

Identify the risks, limitations, and failure modes of LLM systems

Key Points

Six named failure modes: hallucination, non-determinism, injection/jailbreak, data leakage, model drift, automation bias.
Each failure mode has a distinct, non-interchangeable mitigation -- grounding fixes hallucination but not leakage; pinning versions fixes drift but not automation bias.
Confidence is not correctness -- fluent output still requires grounding and verification.
Hallucination mitigations: retrieval grounding, citations, constrained claims, explicit 'I don't know' allowance.
Automation bias (humans over-trusting confident AI output) is countered with transparency and mandatory human validation.

Decision Rules

When: A high-stakes claim is stated fluently and confidently

→Still require grounding and verification -- confidence is not correctness.

When: Mitigating hallucination in a high-stakes answer

→Ground with retrieval, cite sources, and allow 'I don't know' -- not raise temperature or trust stated confidence.

When: Humans start rubber-stamping confident AI output

→Add transparency and mandatory human validation to counter automation bias.

✗ Anti-Patterns to Reject

Treating a fluent, confidently-worded answer as evidence of correctness.
Applying one failure mode's mitigation (e.g., grounding) and assuming it also covers a different failure mode (e.g., data leakage).

ts-ccarp-5.4

Design human-in-the-loop validation for high-stakes agent actions

Key Points

Decide HITL placement by stakes and reversibility, not a general instinct to add oversight.
High-stakes, irreversible actions (financial transactions, account deletion, medical/legal outputs) require pre-execution human approval.
Lower-stakes or reversible actions can run autonomously but benefit from detective review (sampling/auditing).
HITL must be paired with deterministic gating so it cannot be bypassed by the model.
Logging after the fact is detective only -- it is not a substitute for a preventive approval gate.

Decision Rules

When: An action is high-value and hard to reverse (e.g., an automatic refund)

→Require human-in-the-loop approval before execution, gated deterministically.

When: An action is lower-stakes or reversible

→Allow autonomous execution with periodic detective sampling instead of full gating.

When: Confidence is low or the request is out of policy

→Escalate to a human rather than guessing or proceeding.

✗ Anti-Patterns to Reject

Trusting the model's judgment to skip human review 'to keep things fast' on a high-stakes, hard-to-reverse action.
Logging high-stakes actions and moving on instead of gating them with pre-execution approval.

ts-ccarp-5.5

Design for regulatory compliance across GDPR, HIPAA, and FedRAMP

Key Points

GDPR: data minimization, lawful basis, data-subject rights, residency.
HIPAA: PHI safeguards, HIPAA-eligible services, restrict/redact PHI.
FedRAMP: authorized environments and controls for government workloads.
Compliance is a whole-system property -- data handling, access, retention, contracts -- not the model alone.
Anthropic's enterprise features (HIPAA-eligible options, audit logs, retention controls, SSO/SCIM) support but do not automatically satisfy compliance.

Decision Rules

When: A system processes protected health information

→Use HIPAA-eligible services/agreements and restrict/redact PHI -- 'the model is HIPAA-compliant' is not a valid framing.

When: Handling EU personal data

→Apply data minimization, honor data-subject rights, and respect residency requirements.

When: Serving a government workload

→Use authorized environments and controls appropriate to FedRAMP.

✗ Anti-Patterns to Reject

Assuming 'the model is compliant' implies the whole system is compliant.
Treating encryption of the prompt alone as sufficient without addressing access, retention, and contractual controls.

ts-ccarp-5.6

Address ethical AI concerns: bias, fairness, transparency, and accountability

Key Points

Bias/fairness cases belong in the eval set alongside accuracy and safety tests -- evaluate for disparate treatment across groups.
Transparency: disclose AI interaction, cite sources for grounded claims, document decision logic.
AI Fluency accountability model: delegation with description, discernment, and diligence -- verify rather than assume.
Agentic and MCP use carry additional AUP responsibilities beyond the baseline Usage Policy.
Ethics is a testable and monitorable engineering property, not a checklist item handled outside the technical design.

Decision Rules

When: Building an eval set

→Include fairness/bias test cases alongside accuracy and safety cases.

When: A human delegates a task to an agent

→Apply discernment and diligence in checking the result before relying on it -- accountability doesn't disappear because it was delegated.

When: A design uses agentic or MCP capabilities

→Verify it meets the additional AUP responsibilities beyond the baseline Usage Policy.

✗ Anti-Patterns to Reject

Treating bias, fairness, and transparency as a soft, post-hoc checklist separate from engineering.
Assuming delegating a task to an AI removes human accountability for the outcome.