Timezone: »

 
When are equilibrium networks scoring algorithms?
Russell Tsuchida · Cheng Soon Ong

@
in NeurIPS 2022 Workshop on Score-Based Methods »
Event URL: https://openreview.net/forum?id=DEcFxVOE2py »

Principal Component Analysis (PCA) and its exponential family extensions have three components: observed variables, latent variables and parameters of a linear transformation. The likelihood of the observation is an exponential family with canonical parameters that are a linear transformation of the latent variables. We show how joint maximum a-posteriori (MAP) estimates can be computed using a deep equilibrium model that computes roots of the score function. Our analysis provides a systematic way to relate neural network activation functions back to statistical assumptions about the observations. Our layers are implicitly differentiable, and can be fine-tuned in downstream tasks, as demonstrated on a synthetic task.

Author Information

Russell Tsuchida (CSIRO)
Cheng Soon Ong (Data61 and Australian National University)

Cheng Soon Ong is a principal research scientist at the Machine Learning Research Group, Data61, CSIRO, and is the director of the machine learning and artificial intelligence future science platform at CSIRO. He is also an adjunct associate professor at the Australian National University. He is interested in enabling scientific discovery by extending statistical machine learning methods.

More from the Same Authors