Timezone: »

 
Spotlight
Estimating Mutual Information for Discrete-Continuous Mixtures
Weihao Gao · Sreeram Kannan · Sewoong Oh · Pramod Viswanath

Tue Dec 05 03:45 PM -- 03:50 PM (PST) @ Hall C
in Theory »

Estimation of mutual information from observed samples is a basic primitive in machine learning, useful in several learning tasks including correlation mining, information bottleneck, Chow-Liu tree, and conditional independence testing in (causal) graphical models. While mutual information is a quantity well-defined for general probability spaces, estimators have been developed only in the special case of discrete or continuous pairs of random variables. Most of these estimators operate using the 3H -principle, i.e., by calculating the three (differential) entropies of X, Y and the pair (X,Y). However, in general mixture spaces, such individual entropies are not well defined, even though mutual information is. In this paper, we develop a novel estimator for estimating mutual information in discrete-continuous mixtures. We prove the consistency of this estimator theoretically as well as demonstrate its excellent empirical performance. This problem is relevant in a wide-array of applications, where some variables are discrete, some continuous, and others are a mixture between continuous and discrete components.

Author Information

Weihao Gao (UIUC)
Sreeram Kannan (University of Washington)
Sewoong Oh (UIUC)
Pramod Viswanath (UIUC)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors