Skip to main content

12 docs tagged with "Compliance Engineer"

View all tags

Federated analytics: compute only the "statistics," never centralize the raw data — but the guarantee still hinges on DP + secure aggregation

In one sentence that trains a model, this computes statistics. Already deployed the word "federated" does not automatically mean private — keeping data local is only the start; the real privacy guarantee hinges on whether DP and secure aggregation are actually done right; miss either layer and the aggregate result plus multi-round queries can still shake individuals loose.

Fine-tuning-as-a-service privacy: where your fine-tuning data goes, and how fine-tuning erodes alignment (privacy refusals included)

In one sentence how long it's retained, whether it's used for training or human review, whether the resulting fine-tuned model is yours alone — none of this is a single number; verify each vendor's current terms item by item (most vendor docs say fine-tuning inputs/outputs aren't used for training by default and the fine-tuned model is for your use alone — but still check the terms for your tier, your endpoint, your region, and remember they change). Face 2 — fine-tuning itself erodes alignment handing data to a vendor fine-tuning API means you must both verify the data boundary and assume fine-tuning will weaken alignment — including the privacy refusals the model would otherwise make. (Note: Qi et al. is primarily a result about safety-alignment erosion, not a direct data-leak / PII-extraction result — this entry uses it to say "fine-tuning weakens privacy refusals along with the rest," and does not overstate it as "fine-tuning can extract the training data.")

Inference-service data boundary: "they don't train on it" is one cell — verify each, and it expires

In one sentence how long it's retained, how long abuse-monitoring logs are kept, how enterprise vs. consumer tiers differ, whether there's zero data retention (ZDR), who the subprocessors are, which region the data lands in, and whether there's a DPA/BAA. And remember: these terms change, and can be overridden by a legal order. Treating one line — "we don't train on it" — as the whole boundary is the most common operational-phase false security.

LLM watermarking & data provenance: they can mark generated text and detect 'was my data trained on,' but a paraphrase wipes it out

In one sentence one paraphrase with enough budget wipes the signal out. Kirchenbauer et al.'s (ICML 2023) green/red-list watermark gives interpretable p-values from a z-test and needs no model access to detect, but detectability trades against text entropy (low-entropy / short outputs are harder to mark); their follow-up reliability study (ICLR 2024) measured that human and especially LLM-based paraphrase markedly lower detectability, the watermark gets "spread out," detection needs more tokens, and WinMax + SelfHash only partially recover. Conclusion first: a watermark is probabilistic evidence for provenance / forensics, not a strong guarantee — don't read "watermarked" as "can't be removed."

MCP data flow and least collection: when I connect an MCP server, which slice of my private context gets handed over, and who over-collects

In one sentence the MCP spec states plainly that "hosts must obtain explicit user consent before exposing user data to servers," and Anthropic's software directory policy goes further — software "must only collect data from the user's context that is necessary to perform their function" and "must not collect extraneous conversation data, even for logging purposes." Yet in practice a server is routinely granted far more scope than it needs (the entire home directory to serve one folder), credentials concentrate into a single high-value target across servers, and data detours through someone else's infra on a third-party-hosted server. Conclusion first: map which context fields each server actually receives, and whether those fields are what it needs, before you decide to connect it. Treating "only connect official servers" or "one consent prompt is enough" as the whole boundary is the most common operational false security at this layer.

Synthetic-data privacy: replacing real data with synthetic data ≠ anonymous — unless generated under formal DP, and even then a utility trade-off remains

In one sentence synthetic data provides no anonymity by default. Stadler et al. (USENIX Security 2022) showed that any privacy evaluation based on "how similar real and synthetic records look" severely underestimates risk, and that synthetic data is no safer than traditional anonymisation — unless generated under a formal differential-privacy (DP) guarantee; and even with DP, you can't escape a hard privacy↔utility trade-off. Chen et al. (GAN-Leaks, CCS 2020) go further: even when the attacker can only sample from the generator (full black-box — exactly the "publish a synthetic dataset" case), membership inference (MIA) still distinguishes whether a given real record was in the training set above chance, and gets more accurate as overfitting rises. The only thing that gives a bound is formal DP, and ε is not zero.

Unlearning benchmarks and standardized evaluation: turning "did it forget cleanly?" into a regression-able score — and the benchmarks' own blind spots

In one sentence not just "can it still recite it?" but forget quality × retained utility × privacy leakage under attack, three axes at once. And these benchmarks converge on an uncomfortable finding — few methods survive both the utility test and the leakage test (MUSE reports that of eight algorithms, only one avoids severe privacy leakage ⚠️ preprint). But the benchmarks have blind spots of their own: a benchmark is a proxy, and passing it ≠ truly forgotten (this picks up this volume's Unlearning verifiability); whatever it doesn't cover is a corner it can't see.

Verifiability of machine unlearning: you deleted it but can't prove it — model-level unlearning is unverifiable, and "proofs" can be forged

In one sentence the same model parameters can be produced from a different dataset / a different gradient sequence, so a model owner can pass off a fabricated "I unlearned it" proof while actually keeping the record. Alongside it, TOFU (Maini et al., COLM 2024) turns "forget quality" into a measurable benchmark and finds no off-the-shelf method convincingly passes the "forget quality vs. utility" tradeoff. Conclusion first: audit the algorithm / process, not the final weights; "deleted" must be provable, or it's compliance theater.

Verifiable deletion and machine unlearning: "deleted the source record" ≠ "the model forgot" — and proving it forgot is harder still

In one sentence once memorization is in the weights, deleting the source doesn't make it vanish). Machine unlearning aims to make the model "behave as if it never saw that record." Two hard parts: ① how to actually forget (exact unlearning ≈ equivalent retraining, expensive; approximate unlearning is fast but unguaranteed); ② how to prove it forgot (verifiable deletion) — the piece engineering most lacks.