Timezone: »

 
Poster
The Privacy Onion Effect: Memorization is Relative
Nicholas Carlini · Matthew Jagielski · Chiyuan Zhang · Nicolas Papernot · Andreas Terzis · Florian Tramer

Tue Nov 29 02:00 PM -- 04:00 PM (PST) @ Hall J #512
in Poster Session 2 »

Machine learning models trained on private datasets have been shown to leak their private data. Recent work has found that the average data point is rarely leaked---it is often the outlier samples that are subject to memorization and, consequently, leakage. We demonstrate and analyze an Onion Effect of memorization: removing the "layer" of outlier points that are most vulnerable to a privacy attack exposes a new layer of previously-safe points to the same attack. We perform several experiments that are consistent with this hypothesis. For example, we show that for membership inference attacks, when the layer of easiest-to-attack examples is removed, another layer below becomes easy-to-attack. The existence of this effect has various consequences. For example, it suggests that proposals to defend against memorization without training with rigorous privacy guarantees are unlikely to be effective. Further, it suggests that privacy-enhancing technologies such as machine unlearning could actually harm the privacy of other users.

Keywords: privacy · memorization · auditing
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Nicholas Carlini (Google)
Matthew Jagielski (Google)
Chiyuan Zhang (Google Research)
Nicolas Papernot (University of Toronto and Vector Institute)
Andreas Terzis (Google)
Florian Tramer (ETH Zurich)

More from the Same Authors