Timezone: »
Distributed learning of probabilistic models from multiple data repositories with minimum communication is increasingly important. We study a simple communication-efficient learning framework that first calculates the local maximum likelihood estimates (MLE) based on the data subsets, and then combines the local MLEs to achieve the best possible approximation to the global MLE, based on the whole dataset jointly. We study the statistical properties of this framework, showing that the loss of efficiency compared to the global setting relates to how much the underlying distribution families deviate from full exponential families, drawing connection to the theory of information loss by Fisher, Rao and Efron. We show that the "full-exponential-family-ness" represents the lower bound of the error rate of arbitrary combinations of local MLEs, and is achieved by a KL-divergence-based combination method but not by a more common linear combination method. We also study the empirical properties of the KL and linear combination methods, showing that the KL method significantly outperforms linear combination in practical settings with issues such as model misspecification, non-convexity, and heterogeneous data partitions.
Author Information
Qiang Liu (UC Irvine)
Alexander Ihler (UC Irvine)
More from the Same Authors
-
2021 : Temporal-Difference Value Estimation via Uncertainty-Guided Soft Updates »
Litian Liang · Yaosheng Xu · Stephen McAleer · Dailin Hu · Alexander Ihler · Pieter Abbeel · Roy Fox -
2018 Poster: Lifted Weighted Mini-Bucket »
Nicholas Gallo · Alexander Ihler -
2017 Workshop: NIPS Highlights (MLTrain), Learn How to code a paper with state of the art frameworks »
Alex Dimakis · Nikolaos Vasiloglou · Guy Van den Broeck · Alexander Ihler · Assaf Araki -
2017 Poster: Dynamic Importance Sampling for Anytime Bounds of the Partition Function »
Qi Lou · Rina Dechter · Alexander Ihler -
2016 Poster: Learning Infinite RBMs with Frank-Wolfe »
Wei Ping · Qiang Liu · Alexander Ihler -
2015 Poster: Probabilistic Variational Bounds for Graphical Models »
Qiang Liu · John Fisher III · Alexander Ihler -
2015 Poster: Decomposition Bounds for Marginal MAP »
Wei Ping · Qiang Liu · Alexander Ihler -
2013 Workshop: Crowdsourcing: Theory, Algorithms and Applications »
Jennifer Wortman Vaughan · Greg Stoddard · Chien-Ju Ho · Adish Singla · Michael Bernstein · Devavrat Shah · Arpita Ghosh · Evgeniy Gabrilovich · Denny Zhou · Nikhil Devanur · Xi Chen · Alexander Ihler · Qiang Liu · Genevieve Patterson · Ashwinkumar Badanidiyuru Varadaraja · Hossein Azari Soufiani · Jacob Whitehill -
2013 Poster: Scoring Workers in Crowdsourcing: How Many Control Questions are Enough? »
Qiang Liu · Alexander Ihler · Mark Steyvers -
2013 Spotlight: Scoring Workers in Crowdsourcing: How Many Control Questions are Enough? »
Qiang Liu · Alexander Ihler · Mark Steyvers -
2013 Poster: Variational Planning for Graph-based MDPs »
Qiang Cheng · Qiang Liu · Feng Chen · Alexander Ihler -
2012 Poster: Variational Inference for Crowdsourcing »
Qiang Liu · Jian Peng · Alexander Ihler -
2009 Poster: Particle-based Variational Inference for Continuous Systems »
Alexander Ihler · Andrew Frank · Padhraic Smyth -
2006 Poster: Learning Time-Intensity Profiles of Human Activity using Non-Parametric Bayesian Models »
Alexander Ihler · Padhraic Smyth