In distributed, or privacy-preserving learning, we are often given a set of probabilistic models estimated from different local repositories, and asked to combine them into a single model that gives efficient statistical estimation. A simple method is to linearly average the parameters of the local models, which, however, tends to be degenerate or not applicable on non-convex models, or models with different parameter dimensions. One more practical strategy is to generate bootstrap samples from the local models, and then learn a joint model based on the combined bootstrap set. Unfortunately, the bootstrap procedure introduces additional noise and can significantly deteriorate the performance. In this work, we propose two variance reduction methods to correct the bootstrap noise, including a weighted M-estimator that is both statistically efficient and practically powerful. Both theoretical and empirical analysis is provided to demonstrate our methods.
JUN HAN (Dartmouth College)
I am a Ph.D. student in Computer Science at Dartmouth College.
Qiang Liu (Dartmouth College)
More from the Same Authors
2019 Poster: Deep Generative Video Compression »
Salvator Lombardo · JUN HAN · Christopher Schroers · Stephan Mandt
2017 Poster: Stein Variational Gradient Descent as Gradient Flow »
2016 Poster: Learning Infinite RBMs with Frank-Wolfe »
Wei Ping · Qiang Liu · Alexander Ihler
2016 Poster: Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm »
Qiang Liu · Dilin Wang