What Makes a Robotic Failure Legible?

While data has become a vital ingredient for building and improving robot control systems with machine learning, the vast majority of modern datasets commonly used for ML training provide successful task demonstrations alone. However, for building robust and adaptable systems, understanding failures is a necessary component. In this paper, we discuss the challenges and opportunities that arise from collecting a large-scale dataset of robotic manipulation failures. We posit that failures are context-dependent, subjective, and often difficult to attribute, raising fundamental questions about how they should be defined, represented, and annotated within large-scale robotic datasets. Through this lens, we present an annotation philosophy and framework that treats ambiguity as a feature rather than a bug, enabling diverse contributors to capture multiple valid interpretations of robotic failures. Finally, we outline several major open challenges that require future research, such as how to decide what failures are worth recording, or how to incentivize the collection of diverse failures across the research community.