Timezone: »

Representation Costs of Linear Neural Networks: Analysis and Design
Zhen Dai · Mina Karzand · Nathan Srebro

Thu Dec 09 08:30 AM -- 10:00 AM (PST) @ None #None
For different parameterizations (mappings from parameters to predictors), we study the regularization cost in predictor space induced by $l_2$ regularization on the parameters (weights). We focus on linear neural networks as parameterizations of linear predictors. We identify the representation cost of certain sparse linear ConvNets and residual networks. In order to get a better understanding of how the architecture and parameterization affect the representation cost, we also study the reverse problem, identifying which regularizers on linear predictors (e.g., $l_p$ norms, group norms, the $k$-support-norm, elastic net) can be the representation cost induced by simple $l_2$ regularization, and designing the parameterizations that do so.

Author Information

Zhen Dai (The University of Chicago)
Mina Karzand (Toyota Technological Institute at Chicago)
Nathan Srebro (University of Toronto)

More from the Same Authors