ROAR: Unifying Runs across Heterogeneous AI-Driven Research Systems

Each run of an AI-driven research system (ADRS) is an expensive search over a vast solution space, and dependable evaluation requires many runs, making run data both costly to produce and valuable to retain. Yet this data remains fragmented: teams operate in isolation, ADRS frameworks emit results in different formats, and no shared infrastructure exists to aggregate or compare runs across problems and systems. We contribute ROAR, a community repository for ADRS results, addressing two challenges: reconciling heterogeneous ADRS outputs and enabling analytics across runs with different objectives and scoring functions. We achieve this through a relational schema and parser layer that normalize ADRS outputs into a common format. From a corpus of more than 900 runs from multiple ADRS, we show how pooled data can reveal properties of problem landscapes that are difficult to observe from isolated runs or teams: runs with identical configurations may converge to different regions of a landscape, many runs realize most gains early, and the effectiveness of strategies for incorporating prior solutions into the search varies substantially across problems. Such cross-cutting insights are difficult to obtain while runs remain siloed; ROAR is the first infrastructure designed to unify them.