Timezone: »

 
AI and the Everything in the Whole Wide World Benchmark
Deborah Raji · Emily Denton · Emily M. Bender · Alex Hanna · Amandalynne Paullada

@
in Dataset and Benchmark Poster Session 3 »
Event URL: https://openreview.net/forum?id=j6NxpQbREA1 »

There is a tendency across different subfields in AI to see value in a small collection of influential benchmarks, which we term 'general' benchmarks. These benchmarks operate as stand-ins or abstractions for a range of anointed common problems that are frequently framed as foundational milestones on the path towards flexible and generalizable AI systems. State-of-the-art performance on these benchmarks is widely understood as indicative of progress towards these long-term goals. In this position paper, we explore how such benchmarks are designed, constructed and used in order to reveal key limitations of their framing as the functionally 'general' broad measures of progress they are set up to be.

[ Chat]  [ OpenReview] 

Author Information

Deborah Raji (UC Berkeley)
Emily Denton (Google)
Emily M. Bender (University of Washington)
Alex Hanna (Google)
Amandalynne Paullada (University of Washington)

More from the Same Authors