Better Predictions, Worse Schedules: Cluster-Scheduling Benchmarks Select Predictors on the Wrong Metric

Learning-augmented cluster schedulers predict each job's duration and run the shortest predicted job first, and the literature selects the predictor by prediction accuracy, which we argue is the wrong criterion. Mean completion time depends only on the order of the jobs, and every longer job that runs ahead of a shorter one costs the difference between their durations, so the few long jobs placed too early decide the total while accuracy metrics average over the many short ones. On production traces from Alibaba and Google, the most accurate predictor in a recent scheduling benchmark is beaten as a scheduler, by 18 and 26 percent in mean completion time, by a less accurate model with the same features and architecture trained on the completion-time cost instead. The gap sits in the longest jobs, where the benchmark's evaluation window cannot see it. Benchmarks should select predictors on the schedules they produce, not on how well they predict.