Timezone: »

 
Poster
Rashomon Capacity: A Metric for Predictive Multiplicity in Classification
Hsiang Hsu · Flavio Calmon

Thu Dec 01 09:00 AM -- 11:00 AM (PST) @ Hall J #906
in Poster Session 5 »

Predictive multiplicity occurs when classification models with statistically indistinguishable performances assign conflicting predictions to individual samples. When used for decision-making in applications of consequence (e.g., lending, education, criminal justice), models developed without regard for predictive multiplicity may result in unjustified and arbitrary decisions for specific individuals. We introduce a new metric, called Rashomon Capacity, to measure predictive multiplicity in probabilistic classification. Prior metrics for predictive multiplicity focus on classifiers that output thresholded (i.e., 0-1) predicted classes. In contrast, Rashomon Capacity applies to probabilistic classifiers, capturing more nuanced score variations for individual samples. We provide a rigorous derivation for Rashomon Capacity, argue its intuitive appeal, and demonstrate how to estimate it in practice. We show that Rashomon Capacity yields principled strategies for disclosing conflicting models to stakeholders. Our numerical experiments illustrate how Rashomon Capacity captures predictive multiplicity in various datasets and learning models, including neural networks. The tools introduced in this paper can help data scientists measure and report predictive multiplicity prior to model deployment.

Keywords: Rashomon effect · Rashomon set · Rashomon capacity · probabilistic classifier. · predictive multiplicity · channel capacity

Author Information

Hsiang Hsu (Harvard University)

I am Hsiang Hsu, a Harvard Ph.D. student working with Flavio Calmon, and also a Meta Fellow. My research interests lie in promoting the interpretability of representations, improving privacy and fairness, and understanding prediction uncertainty in machine learning. I believe these are important issues in modern machine learning when trying to deploy the models in practice.

Flavio Calmon (Harvard University)

More from the Same Authors