To Align or Not To Align: Check Your COMPASS Before You Train

Combining two pretrained models, whether by aligning a vision encoder with a language model or by merging two fine-tuned checkpoints, is a costly bottleneck: compatibility is typically known only after joint training. We propose COMPASS (COMPatibility ASsessment Score), a theoretically grounded compatibility diagnostic that estimates whether a model pair will be compatible prior to training or merging. Building on the observation that representational convergence is most robust at the level of local neighborhoods, COMPASS decomposes model compatibility into two cheap-to-compute axes: a pair-specific ease, defined as the $k$-NN overlap between the two encoders, and a paradigm-level ceiling, capturing the maximum overlap their training paradigms allow. Together, these two axes provide a ranking of candidate model pairs and prescribe how to act on each pair: (i) prioritize pairs that already agree and whose paradigms are compatible, (ii) invest additional compute on pairs with compatible paradigms but low initial agreement, (iii) switch the pre-training paradigm when it cannot support sufficient agreement, or (iv) deprioritize pairs that score poorly on both axes. COMPASS is applicable to both cross-modal alignment and task-arithmetic merging, where the encoder pair is replaced by two fine-tuned checkpoints. Across $72$ cross-modal alignment experiments, and $392$ model merges, our compatibility estimates strongly correlate with downstream success, with Pearson correlations of $0.90$ for image-to-text R@1 on Flickr30k, $0.91$ for image-to-text R@1 on COCO, and $0.70$ for mean retention across $8$ image-classification merging tasks.