Trucks Don’t Mean Trump: Diagnosing Human Error in Image Analysis

Humans often use images to make high-stakes decisions. We propose a machine learning approach to analyze the ways in which they err in doing so, leveraging a unique dataset of 16,135,392 human predictions of whether a neighborhood voted for Donald Trump or Joe Biden in the 2020 US election, based on a Google Street View image. We show that by training a machine learning estimator of the Bayes optimal decision for each image, we can provide an actionable decomposition of human error into bias, variance, and noise terms and identify specific features (like pickup trucks) which lead humans astray.