Skip to main content

Real incidents & landmark demonstrations (index)

The mechanisms the entries describe aren't hypothetical. This page gathers the concrete, public, verifiable events scattered across them into one index, split two ways:

  • A · Real production incidents — actually happened to users / systems, with a vendor post-mortem or court record.
  • B · Landmark attack demonstrations — peer-reviewed or publicly reproducible attacks that prove "this class of leak really can be done" and shaped this theme's threat model, but are not "incidents."

Keeping the two apart is itself a privacy discipline: don't inflate "proven feasible in the lab" into "leaked in production," and don't dismiss something just because it "hasn't hit the headlines yet." Each gives one line of fact + the leak surface it trips + the entry that unpacks it + a primary source.

About the numbers

The absolute figures below (hit rates, share affected, window length) are all bound to their specific event / experimental setup — textbook cases only, not general probabilities, and not to be transplanted to your endpoint. Sources are stamped 2026-06; verify against the current version before citing.

A · Real production incidents​

  • EchoLeak (CVE-2025-32711 · Microsoft 365 Copilot · 2025) — A crafted email slipped injected instructions into Copilot's RAG context and zero-click exfiltrated the user's private data over an external link; the attack chained past Microsoft's XPIA injection classifier, link redaction, and CSP, and was ultimately fixed server-side by Microsoft. Trips: untrusted content → agent tool / rendering-surface exfiltration. Unpacked in: Agent tool exfiltration · Primary: Aim Labs write-up, MSRC CVE-2025-32711.

  • CamoLeak (CVE-2025-59145 · GitHub Copilot Chat · 2025) — Injected instructions used GitHub's own trusted Camo image proxy to exfiltrate private source code character-by-character encoded into image requests, bypassing CSP; GitHub's fix was to turn off image rendering in Copilot Chat outright. Trips: trusted-domain + rendering-surface exfiltration. Unpacked in: Agent tool exfiltration · Primary: Legit Security write-up.

  • ChatGPT cross-user data bleed (OpenAI · 2023-03) — A server-side change made redis-py under Asyncio return wrong data with small probability per connection: roughly 1.2% of active Plus users, over a ~9-hour window, had their name / email / billing address / last four credit-card digits seen by another active user, plus others' conversation titles / first messages became visible. The fix added redundancy checks to ensure returned data matched the requester. Trips: shared cache + concurrency race + missing ownership check → cross-tenant bleed. Unpacked in: Cross-session memory bleed · Primary: OpenAI official post-mortem.

  • NYT v. OpenAI data-preservation order (from 2025-05) — In the litigation, the Southern District of New York at one point ordered OpenAI to preserve and segregate "all output log data that would otherwise be deleted," covering consumer ChatGPT and the API; that obligation ended around late 2025-09 and was subsequently terminated, with OpenAI resuming its standard practice (per the company, deleted conversations are typically removed from systems within ~30 days). It confirms: the data you assume is "deleted / auto-deleted on expiry" can be frozen by an external legal order. Trips: deletion / retention overridden by law. Unpacked in: Inference-service data boundary, Persistent agent-memory privacy. (Timeline synthesized from OpenAI's official statements and public court reporting; verify the current ruling before citing.)

  • ⚠️ o3 / o4-mini photo reverse-geolocation craze (2025-04) — After the new models shipped, users found frontier multimodal models startlingly good at inferring a city, landmark, even a specific restaurant or bar from an ordinary photo, and it went viral; reporting stressed that it uses "no metadata, no GPS — just what's in the image," lowering the bar for doxxing. ⚠️ Honest labeling: this is news reporting (secondary source), and the same report notes it's not reliably precise (the older GPT-4o was sometimes more accurate) — so this book only says, qualitatively, "the capability is deployed and widely used," and cites no vendor-level pinpoint-accuracy number. Trips: inference-time inference of a hidden sensitive attribute (location) from image content. Unpacked in: VLM geolocation inference · Primary: TechCrunch report (secondary).

B · Landmark attack demonstrations / empirical results​

These aren't "incidents" — they're peer-reviewed or publicly reproducible attack demonstrations proving "this class of leak really can be done." All absolute numbers are bound to their own experimental setup.

  • Text training-data extraction, carried onto production models (Nasr et al. · 2023) — A prompt that makes the model "repeat a word forever" induces aligned production models like ChatGPT to diverge and emit training data verbatim (including real PII), raising the extractable-data yield by roughly 150×. It pushed "training-data extraction" from "the lab, against GPT-2" to "holds against closed-source production models online too." Proves: alignment and the chat wrapper don't stop memorized data already in the weights from being extracted. Unpacked in: Training-data extraction · Primary: Nasr et al., Scalable Extraction, arXiv 2311.17035.

  • Diffusion models regurgitate training images (Carlini et al. · USENIX Security 2023) — A generate-and-filter pipeline extracted training images from widely used open text-to-image models like Stable Diffusion (about 94 under a strict criterion, ~109 near-copies under a loose one), spanning real people's photos to trademark logos, and showed diffusion models are less private than GANs. Proves: "generative models only draw new images, they don't emit originals" is wrong. Unpacked in: Multimodal training-image extraction · Primary: Carlini et al., Extracting Training Data from Diffusion Models (USENIX Security 2023).

  • Embedding inversion back to near-original text (Morris et al. · EMNLP 2023, Outstanding Paper) — vec2text iteratively "corrects then re-embeds" to converge on a target vector, reconstructing 32-token text from GTR-base embeddings at ~92% exact match, and recovered real names from clinical-record embeddings. Proves: "vectorized = anonymized" is wrong; a stored vector is a recoverable copy of sensitive data. Unpacked in: Embedding inversion · Primary: Morris et al., Text Embeddings Reveal (Almost) As Much As Text (EMNLP 2023).

  • KV-cache side channel reconstructs another user's prompt (PromptPeek · NDSS 2025) — On multi-tenant serving frameworks like SGLang (radix-tree KV-cache + longest-prefix-match scheduling), cache hit / miss timing reconstructs another user's prompt token-by-token: up to ~99% with a known template, ~95% with no background (⚠️ the numbers are bound to its tested framework and setup, not to be extrapolated). It pushes the prompt-cache side channel from "detect reuse" to "reconstruct the plaintext." Proves: encryption doesn't stop timing / cache side channels — the boundary has to be drawn at the deployment layer. Unpacked in: Inference-time side channels · Primary: I Know What You Asked (PromptPeek, NDSS 2025).

  • Reconstructing a sensitive genotype from a model (Fredrikson et al. · USENIX Security 2014) — From a personalized warfarin-dosing model plus partial demographics, the attack infers a patient's sensitive genotype. It's one of the earliest end-to-end real case studies of model inversion / attribute inference. Proves: a model's outputs / confidences themselves can be used to reconstruct training-related sensitive attributes. Unpacked in: Model inversion & attribute inference · Primary: Fredrikson et al., Privacy in Pharmacogenetics (USENIX Security 2014).

How to use this index

First read the mechanism + how to defend in the matching entry, then come back here and treat it as a "crime-scene photo" — what the crash looks like in the real world, and what it cost, makes the case for "why this baseline can't be skipped" better than any warning (cross-reference the Minimum privacy baseline). Two reminders: a vendor absent from production incidents isn't risk-free, just without a public post-mortem; and the absolute numbers in the demonstrations are each bound to their own setup — don't transplant them to your endpoint.