Gradient Conflict in Multimodal Pretraining

Early-fusion multimodal models train text and image tokens through a single set of weights, so both modalities compete for every parameter update. Low-rank optimizers reduce optimizer memory by confining that update to a small subspace of the gradient, but they are validated almost exclusively on text-only training, where only one token distribution is present. We ask whether this restriction aggravates conflict between modality gradients. Pretraining a 400M-parameter Chameleon model on interleaved image-text documents, we measure the angle and the relative magnitude of the text and image gradients throughout training under full-rank AdamW, under a subspace built by sin- gular value decomposition (SVD), and under a control in which the subspace is drawn at random. Conflict is confined to the first few hundred steps and clears in every case, though the low-rank runs pass through a sharper and longer transient than full-rank AdamW before doing so. The relative magnitudes differ: the SVD yields a larger share for image gradients than either full-rank training or a random subspace of the same size, while random projection matches full-rank training. Beyond the transient, restricting the optimizer to a subspace leaves the conflict pattern unchanged and cuts optimizer memory substantially, with both low-rank runs ending 0.17–0.30 higher in validation loss than AdamW at its own tuned learning rate.