Unlearning benchmarks and standardized evaluation: turning "did it forget cleanly?" into a regression-able score — and the benchmarks' own blind spots
In one sentence not just "can it still recite it?" but forget quality × retained utility × privacy leakage under attack, three axes at once. And these benchmarks converge on an uncomfortable finding — few methods survive both the utility test and the leakage test (MUSE reports that of eight algorithms, only one avoids severe privacy leakage ⚠️ preprint). But the benchmarks have blind spots of their own: a benchmark is a proxy, and passing it ≠ truly forgotten (this picks up this volume's Unlearning verifiability); whatever it doesn't cover is a corner it can't see.