Poster
Efficient Approximation of Deep ReLU Networks for Functions on Low Dimensional Manifolds
Minshuo Chen · Haoming Jiang · Wenjing Liao · Tuo Zhao

Tue Dec 10th 10:45 AM -- 12:45 PM @ East Exhibition Hall B + C #216
in Theory -- Hardness of Learning and Approximations »

Deep neural networks have revolutionized many real world applications, due to their flexibility in data fitting and accurate predictions for unseen data. A line of research reveals that neural networks can approximate certain classes of functions with an arbitrary accuracy, while the size of the network scales exponentially with respect to the data dimension. Empirical results, however, suggest that networks of moderate size already yield appealing performance. To explain such a gap, a common belief is that many data sets exhibit low dimensional structures, and can be modeled as samples near a low dimensional manifold. In this paper, we prove that neural networks can efficiently approximate functions supported on low dimensional manifolds. The network size scales exponentially in the approximation error, with an exponent depending on the intrinsic dimension of the data and the smoothness of the function. Our result shows that exploiting low dimensional data structures can greatly enhance the efficiency in function approximation by neural networks. We also implement a sub-network that assigns input data to their corresponding local neighborhoods, which may be of independent interest.

Keywords: Deep Learning · Theory -> Hardness of Learning and Approximations

Author Information

Minshuo Chen (Georgia Tech)
Haoming Jiang (Georgia Institute of Technology)
Wenjing Liao (Georgia Tech)
Tuo Zhao (Georgia Tech)

More from the Same Authors