Replay-Augmented Dynamic Data Pruning for Image Classification

Training deep neural networks requires significant computational resources. This has led to various research topics aimed at reducing training costs. One such method is dynamic data pruning, in which already learned samples are removed during the training process. However, when a sample is removed, the associated decision boundary may drift, which can degrade overall accuracy and can lead to catastrophic forgetting. To account for this, we introduce a replay-augmented dynamic data pruning framework that treats the training process as a stateful sleep-and-reactivation system: the learned samples are temporarily removed and selectively reactivated via a replay mechanism. Five pruning signals, a novel mixed signal, and various replay strategies are evaluated primarily on CIFAR-10, CIFAR-100, and EuroSAT using ResNet-18, with select configurations extended to TinyImageNet using ResNet-50. This framework matches baseline accuracy within seed variance while reducing training compute by up to 53% on accuracy-sensitive configurations, and achieves up to 78% compute reduction when efficiency is prioritized.