Timezone: »

 
Poster
On the number of variables to use in principal component regression
Ji Xu · Daniel Hsu

Wed Dec 11 10:45 AM -- 12:45 PM (PST) @ East Exhibition Hall B + C #234
in Theory -- Regularization »
We study least squares linear regression over $N$ uncorrelated Gaussian features that are selected in order of decreasing variance. When the number of selected features $p$ is at most the sample size $n$, the estimator under consideration coincides with the principal component regression estimator; when $p>n$, the estimator is the least $\ell_2$ norm solution over the selected features. We give an average-case analysis of the out-of-sample prediction error as $p,n,N \to \infty$ with $p/N \to \alpha$ and $n/N \to \beta$, for some constants $\alpha \in [0,1]$ and $\beta \in (0,1)$. In this average-case setting, the prediction error exhibits a ``double descent'' shape as a function of $p$. We also establish conditions under which the minimum risk is achieved in the interpolating ($p>n$) regime.
Keywords: Frequentist Statistics · Regularization · Theory · Theory -> Large Deviations and Asymptotic Analysis; Theory
[ Paper]  [ Poster] 

Author Information

Ji Xu (Columbia University)
Daniel Hsu (Columbia University)

See <https://www.cs.columbia.edu/~djhsu/>

More from the Same Authors