TimeSeriesGym: A Scalable Benchmark for (Time Series) Machine Learning Engineering Agents

We introduce TimeSeriesGym, a scalable benchmarking framework for evaluating Large Language Model (LLM) agents on time series machine learning (ML) engineering tasks. Existing benchmarks lack scalability, focus narrowly on well-structured problems, and rely mainly on outcome-based metrics (e.g., task success rate). Our framework addresses these gaps along two critical dimensions. First, TimeSeriesGym incorporates challenges from diverse sources spanning multiple domains and problem types, targeting both isolated capabilities (e.g., data handling, hyperparameter tuning, research code migration) and their combinations, with tools that enable scalable challenge generation. Second, TimeSeriesGym supports multimodal, skill-based, and holistic evaluation, combining precise quantitative metrics with flexible LLM-based evaluation approaches to balance objective assessment and contextual judgment. Although our initial focus is on time series applications, our framework can be readily extended to other data modalities, broadly enhancing the comprehensiveness and practical utility of agentic AI evaluation. Our experiments demonstrate that even state-of-the-art agents struggle to solve time series ML engineering tasks, highlighting the need for more competent agents and more comprehensive benchmarks to advance LLM-driven agents for solving real-world ML engineering challenges.