SODA: Selective Optimization with Deferred BN Alignment for Efficient Dataset Distillation

Recent AI research has increasingly evolved along two complementary directions: model-centric learning, which improves architectures and training algorithms, and data-centric learning, which improves the quality and compression of training data. Within data-centric learning, dataset distillation on large-scale datasets has attracted growing attention, with decoupled distillation methods like SRe$^2$L, G-VBSM, LPQLD, FADRM have emerged as a representative paradigm. These methods typically synthesize compact datasets by matching the Batch Norm statistics of a teacher network, using BN alignment as an effective training objective. However, modern networks often contain many BN layers, and enforcing alignment at every layer introduces substantial computational redundancy. More importantly, we observe that not all BN layers provide equally informative supervision or consistent performance gains. Motivated by this, we propose SODA, a novel Selective Optimization with Deferred BN Alignment framework for efficient dataset distillation. SODA selectively identifies and optimizes informative BN alignment objectives while deferring unnecessary or low-value alignment computations, substantially improving both distillation efficiency and synthetic data quality. We further provide a theoretical analysis explaining why selective and deferred BN alignment can simultaneously reduce optimization cost and improve generalization. Extensive experiments across multiple datasets of CIFAR-100, Tiny-ImageNet, ImageNet-1K and its subsets thereof demonstrate that SODA achieves state-of-the-art performance while offering significantly improved computational efficiency over existing BN-matching-based dataset distillation methods, surpassing FADRM+ by +1.6\% on ImageNet-1K IPC=10 under ResNet-18 while delivering a $1.54\times$ speedup.