PII Handling and Data-Leakage Prevention

A Claude application is still an ordinary system that happens to include a model, and it must uphold ordinary data-protection discipline around that model rather than treating the LLM as exempt from it.

PII Handling

Minimize what personal data is sent to the model at all, and where possible, redact or tokenize sensitive identifiers before they reach the model. Only send data that policy actually permits sending -- "the model can handle it securely" is not a substitute for not sending unnecessary PII in the first place.

Data-Leakage Prevention

Guard against the model or its tools exposing:

• Secrets (API keys, credentials)
• Other users' data
• Internal system details (system prompts, internal tool schemas, infrastructure specifics)

The key architectural control is scoping: tool access should be restricted so the agent cannot read data the current user shouldn't see in the first place -- this prevents leakage structurally rather than relying on the model to decline to repeat something it was never supposed to have access to. Secrets must never be placed in prompts or logs, since anything in a prompt or log is a leakage surface.

The Confused-Deputy Mechanism Behind Most Leakage

A huge share of real-world data-leakage incidents share one underlying mechanism, worth naming precisely: the confused-deputy problem. A confused-deputy vulnerability occurs when an agent that legitimately holds *broader* privilege than the end user acts on an instruction that came from a *lower*-privilege or untrusted source -- injected content in a fetched web page, a tool result, or a request from a user who shouldn't have access to a given resource in the first place. The agent is "confused" into spending its own higher privilege on behalf of a party who never actually had that privilege themselves.

Concretely: a support agent runs with a service account that can read any customer's billing record (broad privilege, needed so it can serve whichever customer asks). A request -- or an injected instruction hidden in content the agent reads -- asks it to pull *a different* customer's billing history. If the agent has no check tying the privilege it's about to exercise back to what the actual requester is entitled to, it happily uses its own broad service-account privilege on behalf of a party who has no right to that data. The agent didn't lack a credential; it had exactly the credential the attack needed, and nothing stopped it from applying that credential on the wrong party's behalf.

This is the precise mechanism that connects prompt injection (7.1) to over-broad tool access (7.2's least-privilege material): an agent with narrowly-scoped tools has far less to be "confused" into misusing than one with broad access, even when both face the identical injected instruction or the identical illegitimate request. Narrow scoping doesn't make the agent smarter about which instructions to trust -- it simply leaves less privilege sitting around for a successful deception to spend. The fix is the same one least privilege always prescribes: scope the agent's effective privilege to what the *specific caller* is entitled to, not to what the *service account* is capable of, so the agent can never do more on a user's behalf than that user could do directly.

The CIA + Privacy Framing

End to end, a Claude system must maintain authentication, authorization, confidentiality, privacy, and integrity -- the same five properties any secure system must maintain. The LLM is one component inside an otherwise ordinary secure system; it does not replace or override the need for standard access control, encryption, and integrity checks elsewhere in the pipeline.

Common exam traps

• Assuming that keeping PII out of the final visible answer is sufficient. Leakage can occur through logs, intermediate tool calls, or over-scoped data access, not just through the model's final response text.
• Treating the model as a security boundary. Scoping data access at the tool/API layer -- before the model ever sees the data -- is the reliable control; hoping the model declines to leak data it was needlessly given is not.
• Assuming a confused-deputy failure means the agent's credentials were stolen or misconfigured. The credentials are exactly right for the agent's normal job -- the failure is that the agent applied them on behalf of a party who was never entitled to that privilege in the first place.