Privacy map
The sidebar organizes by volume (the reading spine). This page adds two query axes: find entries by technique board, or look up by threat → mitigation — "which entries cover this leak surface, and what residual risk remains."
Want a faster start? Four quick-use axes: new readers enter via The essential 12; before shipping, walk the 10-item Minimum privacy baseline; for the real-world cost, read Real incidents & landmark demonstrations; to understand why it leaks, read by root cause in the Mechanism index.
On coverage
This theme is deep-not-wide and doesn't chase entry count (see PROPOSAL). All 14 technique boards now have at least one shipped entry, or are covered "running through every entry" — from here we deepen rather than pad the board count. If a _planned_ marker ever reappears, it's a gap called out on purpose, so "looks complete" doesn't hide a blank (exactly the kind of false security this book argues against).
Browse by technique board
The 14 technique boards are the query axis (boundaries in PROPOSAL §6.2). Shipped entries hang under their primary board:
- Inference attacks (MIA / inversion / attribute): Membership inference · Model inversion & attribute inference · Property / distribution inference · VLM geolocation inference · LoRA adapter leakage
- Memorization & training-data extraction: Training-data extraction · Training-data deduplication · Privacy-targeted poisoning · Multimodal training-image extraction
- Differential privacy: DP fine-tuning · DP in-context learning
- Federated learning & secure aggregation: Production-grade DP·FL · Gradient leakage · Secure aggregation · Split learning leakage · Federated analytics
- Machine unlearning & right to be forgotten: Verifiable deletion and machine unlearning · Unlearning verification
- Privacy-preserving computation (HE / MPC / TEE): Trusted execution environments · Homomorphic encryption / MPC · Confidential inference · On-device inference
- RAG & agent privacy: Multi-tenant RAG retrieval leakage · Cross-session memory bleed · Agent tool-use exfiltration · Embedding inversion · MCP data flow & least-collection · Multi-agent internal-channel leakage · Computer-use screen capture
- Inference-service privacy: Inference-service data boundary · Inference-time side channels · Fine-tuning-as-a-service privacy
- Context-surface privacy: Context-surface privacy (system prompt / conversation context / tool-result leakage) · Reasoning-trace leakage (sensitive data carried in the reasoning trace / thinking)
- PII detection, redaction & synthetic data: PII regurgitation (training-corpus PII reproduced in conversation, the limits of scrubbing) · Synthetic-data privacy
- Model extraction & stealing: Model extraction & stealing (clone behavior / solve some parameters via the query API)
- Governance & compliance: LLM watermarking & provenance (content provenance / "was it trained on," and the limits of paraphrase removal); also runs through each entry's "Compliance mapping" section, index in Volume 6
- Data lifecycle & data governance: Data lifecycle & deletion propagation (cross-store propagation of deletion requests) · Persistent memory privacy & retention (product memory persisted by design / deletion boundary)
- Privacy evaluation & auditing: Quantifying memorization & auditing (canaries / exposure to quantify memorization) · DP auditing (empirically lower-bound the claimed ε) · Unlearning benchmarks (did it really forget) · Agent privacy eval (AgentDojo); also runs through each entry's "Minimal testable assertions" block
Threat → mitigation matrix
Look up by "the leak surface you worry about": which entries cover it, and what they don't solve.
| Threat (what you worry about) | Entries that cover it | Residual risk / what's not solved |
|---|---|---|
| Private data memorized and reproduced verbatim | Training-data extraction · Training-data deduplication | dedup ≠ elimination (a rare once-seen sample is still memorized), DP ≠ zero leakage, output filtering is cat-and-mouse |
| Generative / vision models regurgitating training images (faces / medical / copyright, near-copies) | Multimodal training-image extraction | driven mostly by duplicated images, dedup reduces not eliminates, a near-copy still infringes / leaks, similarity filtering is cat-and-mouse |
| Deciding whether someone is in the training set (membership itself is sensitive) | Membership inference · DP fine-tuning | average accuracy hides the tail, reducing overfitting has no formal guarantee, watch ε |
| Publishing a LoRA / adapter delta, assuming "public base + small delta" keeps the data safe | LoRA adapter leakage | the public base is a free reference, the delta concentrates the fine-tune fingerprint (0.775 AUC ⚠️ preprint), merging ≠ erasure, parameter-efficient ≠ private |
| DP is on, but the claimed ε may not hold (an implementation / accounting bug silently guts it) | DP auditing | auditing gives a lower bound not an upper bound, a loose bound ≠ correct impl, black-box is looser than white-box, re-audit each version with a strong enough attack |
| Rebuilding what a training sample "looks like" / inferring a sensitive attribute from confidence | Model inversion & attribute inference | inversion gives a class representative not a verbatim image, attribute inference includes population stats (can't reach zero), confidence is the fuel |
| An ordinary photo reverse-geolocated by a multimodal model (doxxing) | VLM geolocation inference | stripping EXIF ≠ anonymized (the cues live in the image content), block-level absolute accuracy still limited but climbing, the bias toward less-developed regions is luck not protection |
| Exercising the right to be forgotten / deletion while the model still remembers / how to prove deletion | Machine unlearning · Unlearning verification | output suppression ≠ true deletion, model-level unlearning is unverifiable and proofs are forgeable, MIA not firing ≠ truly forgotten |
| Ran an unlearning algorithm, but don't know how cleanly it forgot / how to score it | Unlearning benchmarks | passing a benchmark ≠ truly forgotten, most methods fail the utility × leakage joint bar, out-of-coverage is a blind spot, metrics can be gamed (⚠️ MUSE preprint) |
| Vector store / retrieval leakage: embedding inversion, cross-tenant bleed | RAG retrieval leakage · Embedding inversion | vectorized ≠ anonymized (embeddings invert back to text, even verbatim), post-retrieval filtering leaves a window, cache / memory boundary bugs |
| Multiple users sharing memory / cache / session, A's data bleeding to B | Cross-session memory bleed | isolation is the system's job not the model's, races are hard to test exhaustively, persistent memory enlarges the window, third-party cache can't be fully audited |
| What a product memory feature persists across sessions by design, and whether I can delete it | Persistent memory privacy & retention | turning the toggle off ≠ deleting what's stored, deletion must fan out to every copy, a legal hold can freeze deletion, backend copies can't be fully audited |
| An agent, once injected, sending private context out through a tool | Agent tool-use exfiltration | instruction-based defenses unreliable, allowlist / human confirmation can be routed around, multiple tools compound it, robust injection defense unsolved |
| After connecting an MCP server, which of my context is handed over and who over-collects | MCP data flow & least-collection | "only official servers are safe" is wrong, consent isn't one-time, least-collection can't govern what happens after handoff, credential concentration widens the blast radius |
| Private data flowing through internal channels (inter-agent messages / shared memory) during multi-agent collaboration | Multi-agent internal-channel leakage | auditing only the external output misses much of it, a stronger model may leak more, internal channels need redaction too (figures all ⚠️ preprint / workshop) |
| A computer-use agent's screenshot pulls the whole screen (incl. unrelated / sensitive / other people's data) into the model | Computer-use screen capture | it captures the whole screen not the task tile, a clean / isolated desktop + human takeover is the boundary, screen-sharing leaks third parties, injection is a separate entry |
| Cloud provider / co-tenant seeing the prompt or weights | Confidential inference · TEE · HE·MPC | not verifying attestation = useless, root of trust is still the chip vendor + side channels, the crypto route is too expensive |
| Assuming on-device inference (the prompt never leaves) means zero egress | On-device inference | "on-device" ≠ zero egress, the cloud-fallback portion must be verified via confidential inference, the trust boundary just moves to the device, the on-device capability cost |
| The provider retaining / training on / handing off what you send | Inference-service data boundary | "no training" ≠ "no retention", terms vary by endpoint / feature, a legal hold can override deletion |
| Device data centralized, or shared gradients / updates leaking training data | Production-grade DP·FL · Gradient leakage · Secure aggregation | FL ≠ private (gradients invertible), secure aggregation hides the single point but not the aggregate sum / collusion, DP has an ε cost, empirical measures aren't a formal guarantee |
| System prompt / conversation context / tool results extracted | Context-surface privacy | instruction-style "don't recite" ≠ isolation, extraction detection is cat-and-mouse, the boundary must land on backend authn / secret management / permission-scoped retrieval |
| A reasoning model carrying sensitive data in its reasoning trace / thinking (more thinking, more leakage) | Reasoning-trace leakage | "thinking is internal so it's safe" is wrong, answer-side redaction ≠ thinking-side, more compute leaks more, placeholder-instruction compliance under 1% |
| Pasting real records as few-shot examples into the prompt, extractable / membership-inferable on the example set | DP in-context learning | DP-ICL protects the examples not the query, ε ≠ zero, side channels outside the aggregation still leak, a utility cost |
| Private PII reproduced by the model in conversation | PII regurgitation | scrubbing reduces ≠ eliminates (NER misses), de-identification ≠ de-association, output filtering is cat-and-mouse |
| The model itself cloned by queries / some parameters solved | Model extraction & stealing | defense raises cost ≠ eliminates, stealing amplifies offline MIA / extraction, limiting output hurts usability |
| Deleted from the primary DB, but copies still remain in backups / logs / vector store / derived models | Data lifecycle & deletion propagation | propagation only covers known copies, still present within the backup expiry window, the derived-model copy needs unlearning |
| Wanting to measure "how much I memorized" before release | Quantifying memorization & auditing | a canary is a proxy not ground truth, high exposure ≠ necessarily extractable, memorization rises with scale so re-measure per version |
| Not stealing one record, but inferring the training set's population makeup (fraction of women…) | Property / distribution inference | can't reach zero (statistical reality), DP targets the individual not the distribution, white-box leaks most |
| Poisoning the training set to amplify privacy leakage about others | Privacy-targeted poisoning | data integrity = a privacy problem, an untrusted data source is an amplifier, dedup / DP can dampen it |
| Encrypted, but response timing / length / cache leaks what you discussed | Inference-time side channels | the boundary must reach the deployment / network / cache layer, padding / isolated caches help, an evolving arms race |
| Splitting the model so raw data stays local, assuming that stops leakage | Split learning leakage | intermediate activations invert back to inputs, a malicious server can actively hijack the feature space, local ≠ private |
| Sharing only aggregate statistics / heavy hitters (not raw data), assuming aggregation = safe | Federated analytics | aggregation ≠ anonymization (heavy hitters still need DP), repeated queries accumulate leakage, ε must be accounted per query |
| Handing fine-tuning data to a managed "fine-tuning-as-a-service," worried it's retained / extracted | Fine-tuning-as-a-service privacy | "not training the base model" ≠ "not keeping your tuning data," terms vary by vendor / endpoint, the "can fine-tuning data be extracted" angle has thin evidence (open question) |
| Replacing real data with synthetic data for release / training, assuming it's anonymous | Synthetic-data privacy | synthetic without DP isn't anonymous, similarity evals give false negatives, outlier records stay membership-inferable, DP carries a utility cost |
| Wanting to provenance generated content / check "was my data trained on" | LLM watermarking & provenance | paraphrase / translation wipes output watermarks, low-entropy short text is hard to mark, the trap method needs text sufficiently long × repeated enough |
| Wanting to measure an agent's post-injection exfiltration risk before release | Agent privacy eval (AgentDojo) | the benchmark is a proxy not exhaustive, passing ≠ truly safe, robust injection defense unsolved, new attacks keep appearing |
How to use this map
First locate the threat you worry about most, read that entry's mechanism + buildable recipe + minimal testable assertions, then jump laterally via each entry's "How this differs from neighboring techniques" to connect the whole line of defense. Every entry calls out the corresponding false security — don't just look at "what was used," look at "what it doesn't solve."