Are Evaluation Outcomes Artifacts of Design Choices? Evidence from 1,134 Counterfactual Resume Screening Studies
Nicole Meister ⋅ Hannah Cha ⋅ Mohammed Alsobay ⋅ Alexandra Chouldechova ⋅ Alex Dow ⋅ Hanna Wallach ⋅ Jennifer Wortman Vaughan ⋅ Carlos Guestrin ⋅ Tatsunori Hashimoto ⋅ Solon Barocas
Abstract
Generative AI supports a remarkable diversity of tasks, variation in how those tasks are formulated, and a wide range of input types. This same flexibility also creates many \emph{researcher degrees of freedom} in the design of evaluations, but often underrecognized and consequential, \emph{researcher degrees of freedom} where even seemingly minor design differences can substantially affect findings. We demonstrate this phenomenon through an analysis of 40 papers on gender and racial bias in generative AI resume screening. Although these studies address the same broad research question and rely on similar methods, they reach strikingly different conclusions. To understand this divergence, we adapt recent work on integrative experiment design from the social sciences, and decompose the evaluation designs in these papers into 46 distinct dimensions along which the studies exhibit considerable variation. We then run 1,134 counterfactual evaluations by combinatorially varying the evaluation design choices from these original papers to see how conclusions shift under the reasonable alternatives adopted in other papers. We find that results are remarkably fragile: a different design choice flips the conclusion of the reproduced evaluations as much as 26\% of the time. Further, choices that researchers might treat as functionally equivalent---and not just reasonable alternatives---can result in opposing findings. However, this fragility is not always purely stochastic. In some settings, we can build a predictive evaluation model that predicts the result of unseen evaluation designs with a mean held-out predictive $R^2$ of 0.774, revealing that seemingly fragile evaluation results may still have a learnable structure. Overall, these findings suggest that even rigorous evaluations with reasonable design choices may provide surprisingly limited insight when considered in isolation. We therefore propose \textit{predictive evaluations} and \textit{large-scale retrospective analyses} as a new goal for the field.
Successful Page Load