Localizing Transfer Between Memorization Tasks

A central puzzle in transfer learning is why pre-training on one task can accelerate training or improve performance on another task, and what mechanisms underlie this transfer. In this work, we examine the transfer between memorization tasks of random input-output mappings. Surprisingly, we find that transfer still occurs even when the two tasks do not have common structure: we observe that pre-training on images with completely random pixels leads to an acceleration on natural images. Through ablation experiments, we decompose and localize the transfer into two separate effects: a ''trivial'' magnitude-driven transfer in the last layer, and a ''non-trivial'' structure-driven transfer in the covariance of the other layers. These results advance our understanding of the underlying mechanisms of transfer learning and have potential to lead to principled pre-training strategies.