`

Timezone: »

 
Poster
Evaluating model performance under worst-case subpopulations
Mike Li · Hongseok Namkoong · Shangzhou Xia

Tue Dec 07 08:30 AM -- 10:00 AM (PST) @ None #None
The performance of ML models degrades when the training population is different from that seen under operation. Towards assessing distributional robustness, we study the worst-case performance of a model over all subpopulations of a given size, defined with respect to core attributes $Z$. This notion of robustness can consider arbitrary (continuous) attributes $Z$, and automatically accounts for complex intersectionality in disadvantaged groups. We develop a scalable yet principled two-stage estimation procedure that can evaluate the robustness of state-of-the-art models. We prove that our procedure enjoys several finite-sample convergence guarantees, including dimension-free convergence. Instead of overly conservative notions based on Rademacher complexities, our evaluation error depends on the dimension of $Z$ only through the out-of-sample error in estimating the performance conditional on $Z$. On real datasets, we demonstrate that our method certifies the robustness of a model and prevents deployment of unreliable models.

Author Information

Mike Li (Columbia University)
Hongseok Namkoong (Stanford University)
Shangzhou Xia (Columbia University)

More from the Same Authors

  • 2021 : Robust fine-tuning of zero-shot models »
    Mitchell Wortsman · Gabriel Ilharco · Jong Wook Kim · Mike Li · Hanna Hajishirzi · Ali Farhadi · Hongseok Namkoong · Ludwig Schmidt
  • 2019 : Coffee break, posters, and 1-on-1 discussions »
    Yangyi Lu · Daniel Chen · Hongseok Namkoong · Marie Charpignon · Maja Rudolph · Amanda Coston · Julius von Kügelgen · Niranjani Prasad · Paramveer Dhillon · Yunzong Xu · Yixin Wang · Alexander Markham · David Rohde · Rahul Singh · Zichen (Vincent) Zhang · Negar Hassanpour · Ankit Sharma · Ciarán Lee · Jean Pouget-Abadie · Jesse Krijthe · Divyat Mahajan · Nan Rosemary Ke · Peter Wirnsberger · Vira Semenova · Dmytro Mykhaylov · Dennis Shen · Kenta Takatsu · Liyang Sun · Jeremy Yang · Alexander Franks · Pak Kan Wong · Tauhid Zaman · Shira Mitchell · min kyoung kang · Qi Yang
  • 2019 : Poster Spotlights »
    Hongseok Namkoong · Marie Charpignon · Maja Rudolph · Amanda Coston · Yuta Saito · Paramveer Dhillon · Alexander Markham
  • 2017 Poster: Variance-based Regularization with Convex Objectives »
    Hongseok Namkoong · John Duchi
  • 2017 Oral: Variance-based Regularization with Convex Objectives »
    Hongseok Namkoong · John Duchi