Domain 7: Security and Safety
8.1% of examDefend against prompt injection, jailbreaks, and untrusted input
Key Points
- Prompt injection hides malicious instructions inside data the model processes (a web page, document, email, or tool result) -- the model cannot inherently distinguish data from instructions.
- A jailbreak is input crafted to bypass safety constraints directly (role-play framing, obfuscation, "developer mode").
- There is no syntactic boundary between instruction and data in natural language, unlike SQL injection -- the fix must be architectural, not linguistic.
- Core mitigations: isolate/delimit untrusted content, apply least privilege on tools, back both with guardrails/hooks, validate input and output.
- A larger, more instruction-following model can be MORE susceptible to injection, not less -- capability is not a security control.
Decision Rules
When: A Claude agent summarizes or consumes untrusted content (web pages, documents, tool results)
→Treat it as untrusted, delimit it, and back it with least-privilege guardrails/hooks.
When: Asked how to mitigate prompt injection
→Reject temperature changes, polite prompt caveats, and "use a bigger model" as non-solutions.
When: An injection might slip through content isolation
→Rely on least privilege to bound the blast radius of whatever it triggers.
✗ Anti-Patterns to Reject
- Raising temperature as an injection mitigation.
- A polite system-prompt request ("don't obey injected instructions") treated as an enforceable control.
- Switching to a larger, more capable model to reduce injection risk.
Prevent PII exposure and data leakage
Key Points
- Minimize PII sent to the model; redact or tokenize sensitive identifiers before they reach it; only send what policy permits.
- Guard against the model or its tools exposing secrets, other users' data, or internal system details.
- Scope tool and data access per user so the agent structurally cannot read data it shouldn't see.
- Never put secrets in prompts or logs -- both are leakage surfaces.
- CIA + privacy (authentication, authorization, confidentiality, privacy, integrity) must hold end to end; the LLM is one component, not a replacement for ordinary security practice.
Decision Rules
When: Sending user data to Claude
→Redact or tokenize PII first and send only what's necessary.
When: Scoping an agent's tool access
→Restrict it so it structurally cannot read data the current user shouldn't see, rather than trusting the model to decline.
When: Writing logs or constructing prompts
→Never place secrets or raw PII inside them.
✗ Anti-Patterns to Reject
- Assuming keeping PII out of the final visible answer is sufficient -- leakage can occur via logs or intermediate tool calls.
- Treating the model itself as the security boundary instead of scoping access at the tool/API layer.
Layer guardrails and enforce hard rules with deterministic hooks
Key Points
- Effective guardrails are multiple independent layers -- input filtering, restricted tool permissions, output validation/moderation, monitoring -- so one layer failing doesn't defeat the system.
- Deployments must respect Anthropic's Usage Policy (AUP).
- Secure-by-design: privacy, identity/access management, and least privilege belong in the design from the start, not bolted on later.
- Hooks (PreToolUse, PostToolUse, stop hooks) are deterministic code callbacks that run every time regardless of the model's decision.
- A prompt instruction is probabilistic; a hook is deterministic -- this is why hooks are the answer for "prevent a destructive action."
Decision Rules
When: Asked what makes guardrails "effective"
→Answer multiple independent layers -- not a single strong system-prompt instruction or the newest model.
When: A rule must block a destructive shell command
→Enforce it in a PreToolUse hook or permission rule, not a prompt sentence.
When: Gating task completion on objective criteria
→Use a stop hook as a verification gate (e.g., a passing test or build).
✗ Anti-Patterns to Reject
- Treating a single strong system-prompt instruction as an effective guardrail.
- Using the newest/largest model as a substitute for layered controls.
- Turning off tools entirely for all tasks instead of scoping them to least privilege.
Manage identity, secrets, and API keys correctly
Key Points
- API keys and secrets belong in environment variables or a secrets manager -- never hard-coded, committed to version control, or pasted into prompts/logs.
- Use separate keys per environment (dev/staging/prod); rotate on a schedule; revoke immediately on exposure.
- Authenticate every request (identity) and separately verify authorization/access level (least privilege) before the agent acts on a caller's behalf.
- Log and monitor authorized and attempted access so misuse is detectable.
Decision Rules
When: Storing an API key
→Use an environment variable or secrets manager, scoped per environment, and rotated on a schedule.
When: An agent is about to act on a caller's behalf
→Verify authorization/access level, not just that the caller authenticated successfully.
When: A key is exposed
→Revoke it immediately rather than waiting for the next scheduled rotation.
✗ Anti-Patterns to Reject
- Hard-coding or committing an API key "for the team."
- Pasting a key into the system prompt.
- Assuming authentication alone (verifying who the caller is) is sufficient without a separate authorization check.