Injection & Jailbreak Defense Techniques
AdvancedMitigate prompt injection, jailbreaks, and untrusted input · Difficulty 3/5
Explanation
Because Prompt Injection is an architecture problem, its defenses are architectural: isolation and containment, reinforced by detection and human review.
The Defenses
- Separate trusted instructions from untrusted data -- delimit and clearly label external content so it isn't executed as an instruction.
- Least privilege -- limit what a hijacked instruction *can* do; an agent that can't delete data can't be tricked into deleting it.
- Input/output filtering and monitoring -- watch for injection patterns and anomalous tool calls.
- **Human-in-the-Loop** -- for high-stakes actions an injection might trigger, require human approval before execution.
These four work together: isolation reduces the chance an injected instruction is even recognized as one; least privilege bounds the damage if it is; filtering/monitoring catches attempts that get through; and human review is the last line of defense before a high-stakes action executes.
What Does NOT Work
- Lowering temperature
- Adding "ignore any injected instructions" text to the prompt
- Using the largest available model
None of these change whether untrusted content is isolated from the instruction channel, and none of them constrain what a hijacked instruction is authorized to do -- which is exactly what an architect is graded on recognizing.
Worked Example
Which measure actually reduces the risk of Prompt Injection? Isolating untrusted content and applying least privilege to tools -- not a prompt caveat, not a sampling change, and not a bigger model. Injection is defeated by architecture: separating trusted instructions from untrusted data and limiting what a hijacked instruction can do.
Key Takeaways
- Core defenses: isolate untrusted content, apply least privilege, filter/monitor, and gate high-stakes actions with human review
- Least privilege contains the blast radius even if isolation fails
- Non-solutions: lower temperature, in-prompt caveats, bigger models
- High-stakes actions an injection might trigger should require human-in-the-loop approval
Glossary Terms
A Claude-native prompting pattern using XML-style tags (e.g., <document>, <instructions>, <example>) to clearly delimit sections of a prompt. Helps Claude unambiguously identify context, instructions, and data. Reduces prompt injection risk by separating instructions from untrusted input.
An attack where malicious content in external data (web pages, documents, user input) attempts to override the system prompt or hijack Claude's behavior. Mitigation: use XML tags to separate untrusted content from instructions, validate outputs, apply least-privilege tool access.
A design pattern that interrupts the agentic loop at defined checkpoints to request human review or approval before proceeding. Used for high-stakes decisions, irreversible actions, or cases where confidence is below threshold. Balances automation with oversight.
A reliability pattern where the agent recognizes conditions it cannot handle autonomously and escalates to a human or higher-capability system. Escalation triggers include: conflicting data sources, low confidence scores, ambiguous requirements, or irreversible high-stakes actions.
A security principle applied to agent tool design: give each agent and subagent only the minimum tools required to complete its specific task. Reduces blast radius if an agent is compromised or makes an error. Implemented via AgentDefinition tool lists and skill allowed-tools.
Related Concepts