Governance, Safety & Risk Management

Knowledge of

• The layered defense-in-depth model for guardrails: input validation/sanitization, least-privilege permissions, deterministic controls (hooks/permission rules), output validation, and monitoring, each layer independent so no single failure is catastrophic
• The distinction between a prompt guardrail (probabilistic guidance the model usually follows) and a hook or permission rule (deterministic control that always fires in code)
• Why destructive-action prevention and hard policy limits belong in deterministic controls rather than in a system-prompt sentence
• How each guardrail layer maps to a stage of the request lifecycle: input, permissions, deterministic enforcement, output, and monitoring

Skills in

• Selecting the correct guardrail layer for a given risk (e.g., routing hard limits to a PreToolUse hook rather than a prompt instruction)
• Designing layered guardrails so that input filtering, least-privilege permissions, output validation, and monitoring operate independently and redundantly
• Recognizing when a single-layer guardrail (prompt-only) is insufficient for a hard requirement and recommending a deterministic control instead
• Avoiding the common architecture trap of relying on a single guardrail layer

Knowledge of

• Prompt injection as the signature LLM threat: malicious instructions hidden in untrusted content (web pages, documents, tool results) that the model reads and executes as commands
• Why separating trusted instructions from untrusted data (delimiting/labeling external content) prevents it from being executed as an instruction
• Why least privilege limits the blast radius of a successful injection -- an agent that cannot delete data cannot be tricked into deleting it
• Why lowering temperature, adding "please ignore injected instructions" text, or using a larger/more capable model does not reliably defeat prompt injection

Skills in

• Designing untrusted-content isolation (clear delimiting/labeling) so injected instructions in tool results or documents are not executed as commands
• Applying least-privilege scoping to agent tools and data access specifically to contain the impact of a successful injection
• Adding input/output filtering and monitoring for injection patterns and anomalous tool calls
• Routing high-stakes actions that an injection might trigger through human-in-the-loop approval
• Rejecting non-solutions (temperature, model size, polite prompt caveats) when asked to mitigate injection risk

Knowledge of

• The named LLM failure modes an architect must design around: hallucination, non-determinism, prompt injection/jailbreak, data leakage, model mismatch/drift, and over-reliance/automation bias
• Why confidence is not correctness -- high-stakes claims must be grounded and verified rather than trusted because they are stated fluently
• Why non-determinism means the same input can yield different output, requiring evals and human review on high-stakes paths rather than assuming reproducibility
• How model mismatch/drift across versions changes behavior, requiring pinned versions and re-run evaluations

Skills in

• Mitigating hallucination with retrieval grounding, citations, constrained claims, and explicit "say I don't know" allowances
• Designing around non-determinism with evaluation suites and human review rather than assuming reproducible output
• Scoping data access and validating output to prevent data leakage of PII or secrets
• Pinning model versions and re-running evals to catch behavior changes from model drift
• Countering over-reliance/automation bias with transparency and mandatory human validation on high-stakes outputs

Knowledge of

• The stakes-and-reversibility framework for deciding where a human must approve or review: high-stakes, irreversible actions (financial transactions, account deletion, medical/legal outputs) require human approval before execution
• Detective review as a complementary control: sampling and auditing automated outputs even where each individual action is not gated
• Escalation paths that defer to a human when confidence is low or the request is out of policy
• Human-in-the-loop as a designed control, not a fallback, that must be paired with deterministic gating so it cannot be bypassed

Skills in

• Classifying actions by stakes and reversibility to decide whether they require pre-execution human approval, detective sampling, or no additional review
• Designing escalation paths that defer to humans on low confidence or out-of-policy requests
• Pairing human-in-the-loop review with deterministic gating (hooks/permission rules) so the review step cannot be bypassed by the model
• Implementing sampling-based detective audits for automated outputs that are not individually gated

Knowledge of

• What GDPR, HIPAA, and FedRAMP each govern and the design implications of each (data minimization/lawful basis/residency; PHI safeguards and HIPAA-eligible services; authorized environments/controls for government workloads)
• That compliance is a property of the whole system -- data handling, access, retention, and contracts -- not of the model in isolation
• The compliance-relevant features Anthropic's enterprise offering provides (HIPAA-eligible options, audit logs, data-retention controls, SSO/SCIM) that architects lean on to meet regulatory obligations
• The architect's responsibility to map a regulatory requirement to concrete design choices: what data is collected, where it is processed/stored, who can access it, and how long it is retained

Skills in

• Mapping a regulation (GDPR, HIPAA, or FedRAMP) to concrete design choices about data collection, processing location, storage, access control, and retention
• Selecting HIPAA-eligible services and agreements and restricting/redacting PHI when a system processes protected health information
• Applying data minimization and respecting data-subject rights and residency requirements when designing for GDPR
• Using authorized environments and controls appropriate to FedRAMP for government workloads
• Rejecting the misconception that "the model is compliant" implies the system is compliant

Knowledge of

• Bias and fairness as evaluable properties: outputs should be evaluated for disparate treatment across groups, with fairness cases included in the eval set
• Transparency requirements: being clear with users that they are interacting with AI, citing sources for grounded claims, and documenting how the system makes decisions
• The AI Fluency framing of accountability and diligence -- delegation with description, discernment, and diligence -- that keeps a human accountable for AI-assisted outputs
• Anthropic's Usage Policy (AUP) as the universal standard the design must comply with, including additional responsibilities for agentic and MCP use, and any high-risk-use requirements

Skills in

• Including fairness/bias test cases in the evaluation set to detect disparate treatment across groups
• Designing transparency into the system: disclosing AI interaction, citing sources for grounded claims, and documenting decision logic
• Applying the AI Fluency framework's discernment and diligence to keep a human accountable for verifying AI-assisted outputs rather than assuming correctness
• Verifying that agentic and MCP-based designs meet the additional responsibilities required under Anthropic's Usage Policy
• Treating ethics as testable, monitorable engineering properties rather than a separate post-hoc consideration