On the Pitfalls of Instance-Based Dynamic Curricula

Active learning has provable benefits in learning machine learning models. As such, automatically building an instance-based curriculum to speed up neural network training has been an active area of research. Recent works such as those of Jiang et al. [2019]; Zhou et al. [2020]; Wang et al. [2024b]; Yuan et al. [2025] have shown benefits of using dynamic curricula on standard image classification tasks, where training data is selected adaptively online, based on how the training goes. On the other hand, Wu et al. [2021] demonstrated that static data ordering, designed independently of the actual training run, does not help in learning classifiers for some standard image classification benchmarks (such as CIFAR10 and CIFAR100). In this paper we extend this result to the case of dynamic data selection, still focusing on image classification, showing that, at least on standard benchmarks like CIFAR10, CIFAR100, and ImageNet, the performance of some state-of-the-art instance-based dynamic curriculum selection methods can be traced back to changes in the effective learning-rate schedule due to the subsampling of the data, rather than the careful selection of the exact data points. This highlights the importance of properly controlling all variables in designing experiments and perhaps suggests that standard image classification tasks are not the best benchmarks for studying curriculum learning.