Timezone: »

 
Poster
Data-dependent PAC-Bayes priors via differential privacy
Gintare Karolina Dziugaite · Daniel Roy

Wed Dec 05 07:45 AM -- 09:45 AM (PST) @ Room 517 AB #107
in Wed Poster Session A »

The Probably Approximately Correct (PAC) Bayes framework (McAllester, 1999) can incorporate knowledge about the learning algorithm and (data) distribution through the use of distribution-dependent priors, yielding tighter generalization bounds on data-dependent posteriors. Using this flexibility, however, is difficult, especially when the data distribution is presumed to be unknown. We show how a differentially private data-dependent prior yields a valid PAC-Bayes bound, and then show how non-private mechanisms for choosing priors can also yield generalization bounds. As an application of this result, we show that a Gaussian prior mean chosen via stochastic gradient Langevin dynamics (SGLD; Welling and Teh, 2011) leads to a valid PAC-Bayes bound due to control of the 2-Wasserstein distance to a differentially private stationary distribution. We study our data-dependent bounds empirically, and show that they can be nonvacuous even when other distribution-dependent bounds are vacuous.

Keywords: Learning Theory · Deep Learning · Theory
[ Paper] 

Author Information

Gintare Karolina Dziugaite (Element AI & University of Cambridge)
Daniel Roy (Univ of Toronto & Vector)

More from the Same Authors