In one sentence it turns "an agent doing legit work while leaking the private data it sees" from "we feel it's probably safe" into a pre-release, regression-able privacy eval — measure the injection-driven exfiltration success rate (ASR) for each outbound tool and fold it into the release gate.
In one sentence the privacy boundary can't stop at "the model won't leak on its own" — once I can act (have tools / can render external links) and read untrusted content, private data has an exfiltration channel. Cut it at the architecture level: least-privilege tools, controlled egress, and treat everything I read as untrusted.
In one sentence other open apps, pop-up notifications, background documents, other browser tabs, other people's data on a shared / screen-shared display, even a password manager that happens to be visible. All of it rides the screenshot into the model — ambient private data far beyond what the task needs. This isn't injection (that goes to Agent tool exfiltration) and it isn't MCP data flow (MCP data flow & least collection); it's input-surface over-capture Anthropic's computer use tool and OpenAI's Operator have both shipped, and both flag this surface in their own docs (keep sensitive data off-screen, isolate in a dedicated VM, hand sensitive input to a human, supervise on sensitive sites). The real boundary isn't "the model looks at less on its own" — it's the capture surface: get the unrelated and sensitive stuff off the screen before you screenshot it.
In one sentence what actually ships today is mainly the hardware isolation + remote attestation route — e.g. GPU confidential computing on NVIDIA H100/H200, and Apple Private Cloud Compute as a private cloud architecture with a verifiable transparency log (same mechanism family, different shapes — don't conflate them); the cryptographic route (HE / MPC) is still too expensive at LLM scale. But "confidential" is a marketing word — the real boundary isn't "the vendor says confidential," it's whether you (or your device) verified the attestation, who the threat model covers, and what it still doesn't solve (side channels, trusting the chip vendor, legal orders). This entry takes Volume 1's TEE and HE·MPC foundations down to "can you actually run private inference."
In one sentence the system prompt is not a secret, and shouldn't be a security control (OWASP LLM07:2025) — don't put credentials / business logic / other people's data into the context, and assume it can be extracted.
In one sentence in March 2023 a redis-py concurrency race in ChatGPT let some users see others' chat titles and the first message of a new conversation, and about 1.2% of active Plus users had their name / email / billing address / last-four card digits exposed in a roughly 9-hour window (OpenAI's official postmortem). Conclusion first: cross-session isolation is the system's responsibility, not "I'll be careful not to mix things up" — scope every cache / memory / session by user, with concurrency safety + ownership checks + auditing.
In one sentence govern by "data lineage + deletion propagation," and don't read "deleted from the primary DB" as "deleted" — that's the most common false security here.
In one sentence clip per-sample gradients + add noise) gives a provable property ε is not zero — it's "bounded leakage," not "zero leakage"; and more private usually means lower utility. Don't read "DP was applied" as "private" — look at what ε is, and whether it protects a sample or a user.
In one sentence either it splits the private examples into disjoint subsets and does a noisy aggregation over my outputs across those subsets before answering, or it uses the private data to generate a batch of synthetic examples with an (ε, δ) guarantee that replace the real ones. What it can bound is any single private example's influence on the answer — not me introspecting the prompt. Two boundaries up front: it protects the examples, not the query itself; and ε isn't zero, so out-of-aggregation side channels still leak.
In one sentence insert many independent probes into training, then use how well you can guess "which probes were trained on" to derive an empirical ε lower bound — the privacy you actually deliver is no stronger than that bound. Steinke et al. (NeurIPS 2023 Outstanding Paper) made it runnable in one training run, cheap enough to keep as a regression check. Conclusion first: "we used a DP library" does not mean "the ε holds", and auditing is the only empirical way to turn a claimed ε into an audited ε.
In one sentence an embedding is just another representation of private data, not a de-identified artifact — encrypt it, access-control it, and set its retention as if it could be turned back into the original text; don't relax just because "we only stored vectors."
In one sentence that trains a model, this computes statistics. Already deployed the word "federated" does not automatically mean private — keeping data local is only the start; the real privacy guarantee hinges on whether DP and secure aggregation are actually done right; miss either layer and the aggregate result plus multi-round queries can still shake individuals loose.
In one sentence how long it's retained, whether it's used for training or human review, whether the resulting fine-tuned model is yours alone — none of this is a single number; verify each vendor's current terms item by item (most vendor docs say fine-tuning inputs/outputs aren't used for training by default and the fine-tuned model is for your use alone — but still check the terms for your tier, your endpoint, your region, and remember they change). Face 2 — fine-tuning itself erodes alignment handing data to a vendor fine-tuning API means you must both verify the data boundary and assume fine-tuning will weaken alignment — including the privacy refusals the model would otherwise make. (Note: Qi et al. is primarily a result about safety-alignment erosion, not a direct data-leak / PII-extraction result — this entry uses it to say "fine-tuning weakens privacy refusals along with the rest," and does not overstate it as "fine-tuning can extract the training data.")
In one sentence "sharing gradients" ≠ "private"; FL needs secure aggregation + DP for a real guarantee — don't read "didn't send raw data" as privacy.
In one sentence the remote execution environment is not a trust root — security comes from cryptographic assumptions. The cost is being much slower (HE especially), so today they're used in narrow scenarios, and full private LLM inference is still expensive. Volume 1 covers what they guarantee, where the cost is, and how to choose between them and a TEE.
In one sentence how long it's retained, how long abuse-monitoring logs are kept, how enterprise vs. consumer tiers differ, whether there's zero data retention (ZDR), who the subprocessors are, which region the data lands in, and whether there's a DPA/BAA. And remember: these terms change, and can be overridden by a legal order. Treating one line — "we don't train on it" — as the whole boundary is the most common operational-phase false security.
In one sentence draw the privacy boundary at the deployment / network / cache layer, not just the model layer — otherwise you think encryption makes you safe, while timing and length are speaking on your behalf.
In one sentence one paraphrase with enough budget wipes the signal out. Kirchenbauer et al.'s (ICML 2023) green/red-list watermark gives interpretable p-values from a z-test and needs no model access to detect, but detectability trades against text entropy (low-entropy / short outputs are harder to mark); their follow-up reliability study (ICLR 2024) measured that human and especially LLM-based paraphrase markedly lower detectability, the watermark gets "spread out," detection needs more tokens, and WinMax + SelfHash only partially recover. Conclusion first: a watermark is probabilistic evidence for provenance / forensics, not a strong guarantee — don't read "watermarked" as "can't be removed."
In one sentence the difference between the base and the fine-tuned model is itself the fingerprint of the fine-tune data. An attacker who holds your adapter and that public base can use the public base as a reference frame and run membership inference / extraction amplified by the delta. LoRA-Leak (arXiv 2507.18302, ⚠️preprint) measures it "the delta is small / the base is public" is not privacy — the public base is a free reference in the attacker's hands that raises your members' distinguishability.
In one sentence the MCP spec states plainly that "hosts must obtain explicit user consent before exposing user data to servers," and Anthropic's software directory policy goes further — software "must only collect data from the user's context that is necessary to perform their function" and "must not collect extraneous conversation data, even for logging purposes." Yet in practice a server is routinely granted far more scope than it needs (the entire home directory to serve one folder), credentials concentrate into a single high-value target across servers, and data detours through someone else's infra on a third-party-hosted server. Conclusion first: map which context fields each server actually receives, and whether those fields are what it needs, before you decide to connect it. Treating "only connect official servers" or "one consent prompt is enough" as the whole boundary is the most common operational false security at this layer.
In one sentence: membership inference (MIA) asks a question that looks harmless but isn't — "was this record in my training set or not." It needs no original text (that's the fundamental difference from training-data extraction), only a yes/no; yet when the "yes" is itself sensitive (e.g. "this person is in a particular disease dataset"), that single bit is the leak. It is the core target differential privacy defends against, and the foundation of a whole chain of privacy attacks — extraction, attribute inference, and more — which is why it leads Volume 1.
In one sentence a stolen / cloned model can be attacked offline, repeatedly (membership inference, extraction), amplifying one-time API access into a persistent privacy risk. Conclusion: API access ≠ zero leakage — threat-model "the queries themselves can be used to extract model information."
In one sentence model inversion — using repeated queries + the confidence I emit to rebuild what a class's training sample "looks like" (Fredrikson et al. at CCS 2015 reconstructed recognizable faces from a face-recognition model); and attribute inference — given a person's partial known info + me, inferring their undisclosed sensitive attribute (Fredrikson et al.'s 2014 warfarin-dosing case). Conclusion first: confidence / probability outputs are the fuel, and attribute inference also borrows population-statistics correlations — don't assume "didn't send raw data out" is safe; look at output granularity, whether DP is stacked, and whether a class maps to a single individual.
In one sentence internal channels leak substantially more sensitive data than the external output, and an audit that only looks at the final output misses a large fraction of it. Conclusion first: if your audit boundary stops at "what the model said to the user," it can't see the dominant leakage surface in a multi-agent system — internal channels must also be audited, redacted, and shared on a need-to-know minimum.
In one sentence: in multi-tenant RAG, two intuitions both fail — "vectorized means anonymized" and "filtered by user means isolated." In some studied embedding settings a vector can be inverted back to approximate original text (including PII), so don't treat it as anonymization; retrieval ranks by similarity, not permission, so if the ACL runs after retrieval instead of before it, the most relevant private chunk may come from another tenant; and a single boundary bug in a shared cache / long-term memory can carry one user's private context into another's session.
In one sentence precise street-level accuracy is still limited today — in peer-reviewed numbers, even a purpose-built fine-tuned + chain-of-thought framework only hits about 28.7% within the 1km threshold (ETHAN, PoPETs 2025). So the threat isn't "I can pinpoint every photo"; it's that this capability is already deployed at scale and improving fast. Conclusion first: treating "I stripped the EXIF, so it's safe" as a talisman is false security — the location cues live in the image content itself, so defense has to land on notice-and-restraint before upload, not just on stripping metadata.
In one sentence don't assume "a generative model only paints new images, it won't emit the originals." Image-domain memorization is real, the defenses share the same root as text memorization (dedup + DP + similarity auditing), but the cost is now "near-copy" rather than "verbatim."
In one sentence on-device means "the device doesn't send it out." But "on-device" is a word that's easy to over-read — the real boundary isn't "the vendor says on-device," it's which requests still fall back to the cloud, how much capability you trade away because the on-device model is smaller, and the telemetry / diagnostics / model-download channels. Bottom line first: "on-device" ≠ "nothing leaves the device." What's verifiable is that a given request completes on the local device and its prompt didn't leave the device for that path; but which requests run on-device versus fall back to the cloud is a deployment fact to check, not a model promise. For the fall-back-to-cloud portion, you go back to confidential inference to verify it.
In one sentence how much you can see and delete is decided by the product and its backend. The harder lesson comes from NYT v. OpenAI (2025) govern the product's memory as one more copy of your data, subject to both the product and the law — don't read "I flipped the switch off" as "it's all deleted."
In one sentence Lukas et al., IEEE S&P 2023). Conclusion first: treat PII regurgitation as "reduce + audit + accept it can't be fully eliminated," and don't read "we ran scrubbing" as "no PII" — that's the most common false security here.
In one sentence by buying expired domains for a "split-view," you can control ~0.01% of LAION-400M for ~\$60 (Carlini et al., S&P 2024 — this is a feasibility result, not itself a privacy-leak result). Conclusion first: training-data integrity = a privacy problem; an untrusted data source is a privacy amplifier.
In one sentence DP (bound a single device's / user's influence on the model) + secure aggregation (the server sees only the sum of many devices' updates, not any single one). Real deployments FL ≠ private, watch local DP's ε, and don't mismatch user-level vs sample-level. This is the deployment face of Volume 3 DP fine-tuning at "federated + large-scale production."
In one sentence "what fraction of women / a given ethnicity was this model trained on." Ganju et al. (CCS 2018) did exactly this with a meta-classifier the population makeup of a training set can itself be sensitive (trade secret / group privacy) — don't focus only on protecting a single individual; sometimes the attacker doesn't want any one record at all.
In one sentence insert random canaries into the training set, then use exposure to measure how strongly I prefer each one over random strings — higher exposure means it's memorized harder. Quantifying Memorization (ICLR 2023) goes further and measures audit memorization with canaries + exposure before release, turning "how much got memorized" into a regression-able number — don't say "it probably didn't memorize anything" on a hunch, which is the false security of a missing audit.
In one sentence not really output, not visible, no risk. Wrong. On my side, the thinking and the final answer come out of the same generation channel; products often render the trace straight to the user, write it to logs, or feed it downstream — and the sensitive data you hand me shows up all over that thinking. It can be pulled out by prompt injection aimed at the reasoning, and it leaks into the final answer too. The counterintuitive part do not treat the reasoning trace as a private draft. Once it is shown or logged it is externally observable output; give it the same outbound redaction and access control as the final answer, and don't let "the thinking is internal, so it's safe" become your false security.
In one sentence use secure multiparty computation so the server can compute only "the sum of all clients' updates," never any single update — the single point is hidden, and inversion loses what it stands on. It's communication-efficient and robust to client dropout, and is already used in Google's production FL (Bonawitz et al., MLSys 2019 secure aggregation downgrades "the server is trusted" to "the server only sees the aggregate sum"; but it's not a cure-all — it defends against "seeing a single update," not "the aggregate sum itself leaking" or "parties colluding," so still pair it with DP.
In one sentence this is not private. The Feature-Space Hijacking Attack (Pasquini et al., CCS 2021) shows a malicious server can actively steer the split model into an insecure state and reconstruct the client's private training inputs from the intermediate activations (reconstructing images on MNIST / Omniglot / CelebA); UnSplit (Erdoğan et al., WPES @ CCS 2022) shows that even an honest-but-curious server — knowing only the client architecture, with no active interference — can invert (reconstruction MSE ≈ 0.08–0.15 on MNIST / Fashion-MNIST / CIFAR-10). Don't read "didn't send raw pixels" as privacy.
In one sentence synthetic data provides no anonymity by default. Stadler et al. (USENIX Security 2022) showed that any privacy evaluation based on "how similar real and synthetic records look" severely underestimates risk, and that synthetic data is no safer than traditional anonymisation — unless generated under a formal differential-privacy (DP) guarantee; and even with DP, you can't escape a hard privacy↔utility trade-off. Chen et al. (GAN-Leaks, CCS 2020) go further: even when the attacker can only sample from the generator (full black-box — exactly the "publish a synthetic dataset" case), membership inference (MIA) still distinguishes whether a given real record was in the training set above chance, and gets more accurate as overfitting rises. The only thing that gives a bound is formal DP, and ε is not zero.
In one sentence dedup is a high-return move that cuts memorization / extraction / membership-inference risk a lot; but it lowers frequency and probability, not a formal guarantee — a rare sample appearing once can still be memorized; for a formal guarantee, stack DP.
In one sentence the larger the model, the more often it was duplicated, and the more context you give, the more discoverable memorization there is.
In one sentence the root of trust is the chip vendor (you trust Intel / AMD / NVIDIA's keys and implementation), and TEEs have been repeatedly broken by microarchitectural side channels (the Foreshadow family). It's the load-bearing wall for Volume 5's "confidential inference," so Volume 1 spells out up front what it does and doesn't guarantee.
In one sentence not just "can it still recite it?" but forget quality × retained utility × privacy leakage under attack, three axes at once. And these benchmarks converge on an uncomfortable finding — few methods survive both the utility test and the leakage test (MUSE reports that of eight algorithms, only one avoids severe privacy leakage ⚠️ preprint). But the benchmarks have blind spots of their own: a benchmark is a proxy, and passing it ≠ truly forgotten (this picks up this volume's Unlearning verifiability); whatever it doesn't cover is a corner it can't see.
In one sentence the same model parameters can be produced from a different dataset / a different gradient sequence, so a model owner can pass off a fabricated "I unlearned it" proof while actually keeping the record. Alongside it, TOFU (Maini et al., COLM 2024) turns "forget quality" into a measurable benchmark and finds no off-the-shelf method convincingly passes the "forget quality vs. utility" tradeoff. Conclusion first: audit the algorithm / process, not the final weights; "deleted" must be provable, or it's compliance theater.
In one sentence once memorization is in the weights, deleting the source doesn't make it vanish). Machine unlearning aims to make the model "behave as if it never saw that record." Two hard parts: ① how to actually forget (exact unlearning ≈ equivalent retraining, expensive; approximate unlearning is fast but unguaranteed); ② how to prove it forgot (verifiable deletion) — the piece engineering most lacks.