How Do Training Methods Influence the Utilization of Vision Models?

Not all weights contribute equally to the decision function of neural networks. In fact, the weights of entire layers can sometimes be reset to random values without significantly affecting model decisions. We revisit earlier studies that examined how architecture and task complexity influence this phenomenon and ask: is this phenomenon also affected by the training method?To explore this, we conducted experimental evaluations on a diverse set of ImageNet-1k classification models, keeping the architecture and training data constant but varying the training pipeline. Our findings reveal that the training method strongly influences which layers become critical to the decision function for a given task. For example, improved training regimes and self-supervised training increase the importance of early layers while significantly under-utilizing deeper layers. In contrast, methods such as adversarial training display a different trend.Our preliminary results extend previous findings, offering a more nuanced understanding of the inner mechanics of neural networks.Code: https://after-accept.com