Abstract: This paper studies the properties of solutions to multi-task shallow ReLU neural network training problems. Remarkably, solutions to multi-task problems can be identical to those of kernel methods, revealing a novel connection between neural networks and kernel methods. We focus on the problem of interpolating training data subject to minimizing the sum of squared weights in the network. It is known that the functions that result from such single-task training problems solve a minimum norm interpolation problem in a non-Hilbertian Banach space and are generally non-unique. Moreover, the multiple solutions can be significantly different from each other and can exhibit undesirable behaviors such as making large excursions from the training data. In contrast, the solutions to multi-task training problems are strikingly different. In univariate settings, we prove that the solution to multi-task training is almost always unique and that the solution to each individual task is equivalent to minimum norm interpolation in a Sobolev (Reproducing Kernel) Hilbert space. Moreover, the unique multi-task solution has desirable generalization properties that the single-task solutions can lack. We also provide empirical evidence and mathematical analysis which shows that multivariate multi-task training can also lead to solutions that are related to a minimum $\ell^2$-norm (Hilbert space) interpolation.