Spectral Insights from the Unconstrained Feature Model for Neural Multi-Output Regression

Multi-output regression can be approached either by fitting one joint vector-valued predictor or by training separate univariate predictors for each target coordinate. This holistic-versus-separate choice is classical in the largely non-neural multi-output regression literature, but is less characterized analytically for neural networks, where joint prediction arises naturally through shared features and vector-valued output layers. Because neural networks are highly non-linear and high-dimensional, exact analysis is notoriously difficult. We study two questions using the Unconstrained Feature Model (UFM) as a tractable surrogate for exact analysis: $\textbf{(1)}$ how does joint-output regression compare with independent coordinate-wise regression under regularization? $\textbf{(2)}$ when can target whitening or normalization improve original-scale MSE? Under UFM, the minimal training MSE depends only on the regularization product and the spectrum of the target covariance. This yields two theoretical results: joint-output regression has no larger training MSE than independent coordinate-wise regression under matched regularization, and whitening or normalizing targets can help or hurt original-scale MSE depending on the average target variance. Experiments on neural networks trained with standard weight decay across robotic imitation-learning and autonomous-driving datasets support these theoretical results, with test-MSE trends following the same qualitative pattern. Our results show that simplified analytical surrogates can provide useful guidance for neural multi-output regression.