Timezone: »
It has been observed that machine learning models trained using stochastic gradient descent (SGD) exhibit poor generalization to certain groups within and outside the population from which training instances are sampled. This has serious ramifications for the fairness, privacy, robustness, and out-of-distribution (OOD) generalization of machine learning. Hence, we theoretically characterize the inherent generalization of SGD-learned overparameterized linear regression to intra- and extra-population groups. We do this by proving an excess risk bound for an arbitrary group in terms of the full eigenspectra of the data covariance matrices of the group and population. We additionally provide a novel interpretation of the bound in terms of how the group and population data distributions differ and the effective dimension of SGD, as well as connect these factors to real-world challenges in practicing trustworthy machine learning. We further empirically validate the tightness of our bound on simulated data.
Author Information
Arjun Subramonian (University of California, Los Angeles)
Levent Sagun (Facebook AI Research)
Kai-Wei Chang (UCLA)
Yizhou Sun (UCLA)
More from the Same Authors
-
2022 : Unit Selection: Learning Benefit Function from Finite Population Data »
Ang Li · Song Jiang · Yizhou Sun · Judea Pearl -
2022 : Dissimilar Nodes Improve Graph Active Learning »
Zhicheng Ren · Yifu Yuan · Yuxin Wu · Xiaxuan Gao · Yewen Wang · Yizhou Sun -
2022 : Empowering Language Models with Knowledge Graph Reasoning for Question Answering »
Ziniu Hu · Yichong Xu · Wenhao Yu · Shuohang Wang · Ziyi Yang · Chenguang Zhu · Kai-Wei Chang · Yizhou Sun -
2023 Poster: CARE: Modeling Interacting Dynamics Under Temporal Distribution Shift »
Xiao Luo · Haixin Wang · Zijie Huang · Huiyu Jiang · Abhijeet Gangan · Song Jiang · Yizhou Sun -
2023 Poster: A Pseudo-Semantic Loss for Deep Generative Models with Logical Constraints »
Kareem Ahmed · Kai-Wei Chang · Guy Van den Broeck -
2023 Poster: DesCo: Learning Object Recognition with Rich Language Descriptions »
Liunian Li · Zi-Yi Dou · Nanyun Peng · Kai-Wei Chang -
2023 Poster: Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models »
Pan Lu · Baolin Peng · Hao Cheng · Michel Galley · Kai-Wei Chang · Ying Nian Wu · Song-Chun Zhu · Jianfeng Gao -
2023 Poster: AVIS: Autonomous Visual Information Seeking with Large Language Models »
Ziniu Hu · Ahmet Iscen · Chen Sun · Kai-Wei Chang · Yizhou Sun · Cordelia Schmid · David Ross · Alireza Fathi -
2023 Poster: Amazon-M2: A Multilingual Multi-locale Shopping Session Dataset for Recommendation and Text Generation »
Wei Jin · Haitao Mao · Zheng Li · Haoming Jiang · Chen Luo · Hongzhi Wen · Haoyu Han · Hanqing Lu · Zhengyang Wang · Ruirui Li · Zhen Li · Monica Cheng · Rahul Goutam · Haiyang Zhang · Karthik Subbian · Suhang Wang · Yizhou Sun · Jiliang Tang · Bing Yin · Xianfeng Tang -
2023 Poster: Towards a Comprehensive Benchmark for FPGA Targeted High-Level Synthesis »
Yunsheng Bai · Atefeh Sohrabizadeh · Zongyue Qin · Ziniu Hu · Yizhou Sun · Jason Cong -
2022 Spotlight: Improving Multi-Task Generalization via Regularizing Spurious Correlation »
Ziniu Hu · Zhe Zhao · Xinyang Yi · Tiansheng Yao · Lichan Hong · Yizhou Sun · Ed Chi -
2022 Workshop: New Frontiers in Graph Learning »
Jiaxuan You · Marinka Zitnik · Rex Ying · Yizhou Sun · Hanjun Dai · Stefanie Jegelka -
2022 Poster: Improving Multi-Task Generalization via Regularizing Spurious Correlation »
Ziniu Hu · Zhe Zhao · Xinyang Yi · Tiansheng Yao · Lichan Hong · Yizhou Sun · Ed Chi -
2022 Poster: On the Discrimination Risk of Mean Aggregation Feature Imputation in Graphs »
Arjun Subramonian · Kai-Wei Chang · Yizhou Sun -
2022 Poster: Semantic Probabilistic Layers for Neuro-Symbolic Learning »
Kareem Ahmed · Stefano Teso · Kai-Wei Chang · Guy Van den Broeck · Antonio Vergari -
2022 Poster: GStarX: Explaining Graph Neural Networks with Structure-Aware Cooperative Games »
Shichang Zhang · Yozen Liu · Neil Shah · Yizhou Sun -
2022 Poster: Controllable Text Generation with Neurally-Decomposed Oracle »
Tao Meng · Sidi Lu · Nanyun Peng · Kai-Wei Chang -
2022 Poster: Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering »
Pan Lu · Swaroop Mishra · Tanglin Xia · Liang Qiu · Kai-Wei Chang · Song-Chun Zhu · Oyvind Tafjord · Peter Clark · Ashwin Kalyan -
2020 Poster: Automatic Perturbation Analysis for Scalable Certified Robustness and Beyond »
Kaidi Xu · Zhouxing Shi · Huan Zhang · Yihan Wang · Kai-Wei Chang · Minlie Huang · Bhavya Kailkhura · Xue Lin · Cho-Jui Hsieh -
2020 Poster: Learning Continuous System Dynamics from Irregularly-Sampled Partial Observations »
Zijie Huang · Yizhou Sun · Wei Wang -
2019 Workshop: Science meets Engineering of Deep Learning »
Levent Sagun · Caglar Gulcehre · Adriana Romero Soriano · Negar Rostamzadeh · Nando de Freitas -
2019 : Welcoming remarks and introduction »
Levent Sagun · Caglar Gulcehre · Adriana Romero Soriano · Negar Rostamzadeh · Nando de Freitas -
2019 Workshop: Graph Representation Learning »
Will Hamilton · Rianne van den Berg · Michael Bronstein · Stefanie Jegelka · Thomas Kipf · Jure Leskovec · Renjie Liao · Yizhou Sun · Petar Veličković -
2019 Poster: Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias »
Stéphane d'Ascoli · Levent Sagun · Giulio Biroli · Joan Bruna -
2019 Poster: Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networks »
Difan Zou · Ziniu Hu · Yewen Wang · Song Jiang · Yizhou Sun · Quanquan Gu -
2016 Poster: Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings »
Tolga Bolukbasi · Kai-Wei Chang · James Y Zou · Venkatesh Saligrama · Adam T Kalai -
2016 Poster: A Credit Assignment Compiler for Joint Prediction »
Kai-Wei Chang · He He · Stephane Ross · Hal Daumé III · John Langford