DP in-context learning: put differential privacy on the private examples in your prompt, not on training
In one sentence either it splits the private examples into disjoint subsets and does a noisy aggregation over my outputs across those subsets before answering, or it uses the private data to generate a batch of synthetic examples with an (ε, δ) guarantee that replace the real ones. What it can bound is any single private example's influence on the answer — not me introspecting the prompt. Two boundaries up front: it protects the examples, not the query itself; and ε isn't zero, so out-of-aggregation side channels still leak.
Synthetic-data privacy: replacing real data with synthetic data ≠ anonymous — unless generated under formal DP, and even then a utility trade-off remains
In one sentence synthetic data provides no anonymity by default. Stadler et al. (USENIX Security 2022) showed that any privacy evaluation based on "how similar real and synthetic records look" severely underestimates risk, and that synthetic data is no safer than traditional anonymisation — unless generated under a formal differential-privacy (DP) guarantee; and even with DP, you can't escape a hard privacy↔utility trade-off. Chen et al. (GAN-Leaks, CCS 2020) go further: even when the attacker can only sample from the generator (full black-box — exactly the "publish a synthetic dataset" case), membership inference (MIA) still distinguishes whether a given real record was in the training set above chance, and gets more accurate as overfitting rises. The only thing that gives a bound is formal DP, and ε is not zero.