FUSA-Net: Notation-Aware Cross-Modal Retrieval for Fine-Grained Audio-Score Alignment

Cross-modal music retrieval (matching audio recordings against their corresponding sheet music) is hindered by the modality gap: audio and score embeddings tend to segregate into structurally distinct regions of the shared latent space even when explicitly trained for alignment. We present FUSA-Net, addressed through three interdependent contributions. PDMX-FUSA is a ~291,670-example public-domain dataset of measure-aligned audio-score-MIDI-feature tuples. A two-phase encoder selection methodology evaluates 40 visual-acoustic encoder pairs and finds a clear qualitative association between CCA mean correlation and contrastive retrieval performance, suggesting CCA as a promising low-cost screening criterion, and revealing that intra-modal discriminative capacity and cross-modal compatibility are largely decoupled. FUSA-Net is a frozen-backbone dual-encoder model with CCA-warm-started residual projection heads and notation-aware multi-task learning. On a genuinely held-out test split of synthesized audio, FUSA-Net achieves a bidirectional Recall@1 of approximately 67.5% and Recall@10 of approximately 93.1%, closely matching validation performance and indicating that the model generalizes rather than overfits to the model-selection process; this nearly doubles the Recall@1 of a zero-shot linear CCA baseline. On a pilot evaluation with real (non-synthesized) piano recordings from a stylistically distinct repertoire, and with no fine-tuning whatsoever, retrieval remains consistently above chance in both directions, an encouraging zero-shot signal that motivates a larger-scale evaluation. We also report a general evaluation-protocol observation we believe is of independent interest to the cross-modal retrieval community, and a concrete set of near-term directions toward a fully validated archival version of this work.