Timezone: »

 
Poster
Two-layer neural network on infinite dimensional data: global optimization guarantee in the mean-field regime
Naoki Nishikawa · Taiji Suzuki · Atsushi Nitanda · Denny Wu

Wed Nov 30 09:00 AM -- 11:00 AM (PST) @ Hall J #922
in Poster Session 3 »

Analysis of neural network optimization in the mean-field regime is important as the setting allows for feature learning. Existing theory has been developed mainly for neural networks in finite dimensions, i.e., each neuron has a finite-dimensional parameter. However, the setting of infinite-dimensional input naturally arises in machine learning problems such as nonparametric functional data analysis and graph classification. In this paper, we develop a new mean-field analysis of two-layer neural network in an infinite-dimensional parameter space. We first give a generalization error bound, which shows that the regularized empirical risk minimizer properly generalizes when the data size is sufficiently large, despite the neurons being infinite-dimensional. Next, we present two gradient-based optimization algorithms for infinite-dimensional mean-field networks, by extending the recently developed particle optimization framework to the infinite-dimensional setting. We show that the proposed algorithms converge to the (regularized) global optimal solution, and moreover, their rates of convergence are of polynomial order in the online setting and exponential order in the finite sample setting, respectively. To our knowledge this is the first quantitative global optimization guarantee of neural network on infinite-dimensional input and in the presence of feature learning.

Keywords: Optimization · Neural Network · mean-field regime · functional data analysis
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Naoki Nishikawa (University of Tokyo)
Taiji Suzuki (The University of Tokyo/RIKEN-AIP)
Atsushi Nitanda (Kyushu Institute of Technology / RIKEN)
Denny Wu (University of Toronto & Vector Institute)

More from the Same Authors