Learning Cross-Modal Representations Between Electrocardiography and Phonocardiography for Low-Resource Cardiac AI

Electrocardiography (ECG) and phonocardiography (PCG) provide complementary information about cardiac function, yet representation-learning methods commonly model these modalities independently, most often fusing them only for a specific downstream task such as classification [4, 5]. This separation is particularly limiting in low-resource clinical settings, where synchronized multimodal data are scarce. We investigate whether synchronized recordings can reveal learnable cross-modal correspondence between the two modalities. Related work has explored cross-modal reconstruction between ECG and PCG [3]; we instead investigate cross-modal correspondence through retrieval and explicitly examine the effect of embedding geometry on similarity-based evaluation. We study the EPHNOGRAM dataset [1, 2], containing 69 synchronized ECG–PCG recordings from 25 subjects, segmented into 22,008 paired five-second windows. Modality-specific ECG and PCG embeddings are extracted and evaluated under subject-level five-fold cross-validation. We first examine conventional embedding similarity and identify an important challenge: PCG embeddings exhibit high similarity even for unrelated samples. Randomly paired PCG embeddings have a mean cosine similarity of 0.969, while their similarity to the global PCG embedding mean is 0.984. These results suggest that raw cosine similarity can be strongly influenced by the geometry of the embedding space and may therefore provide an overly optimistic assessment of cross-modal alignment. We address this issue by centering PCG embeddings using statistics computed exclusively from the training folds and evaluate ECG-to-PCG prediction using linear and nonlinear mappings. After centering PCG embeddings using training-fold statistics, direct ECG-to-PCG reconstruction did not outperform a centered mean baseline (MSE 4.23 vs. 5.33 for linear regression and 5.51 for an MLP). We therefore formulate cross-modal alignment as a shared-space retrieval problem. ECG embeddings are projected toward the PCG representation space and evaluated through retrieval using synchronized ECG–PCG windows as positive pairs. In preliminary five-fold experiments, synchronized pairs achieve a mean cosine similarity of 0.101 compared with 0.005 for randomly paired samples. The synchronized PCG was retrieved as the top-ranked candidate in 7.0% of cases across validation folds, with a mean retrieval rank of 1,746 among thousands of candidates. These results provide preliminary evidence of learnable cross-modal correspondence in our paired ECG–PCG embeddings, while also highlighting the limitations of conventional similarity and direct reconstruction for evaluating this relationship. Our work establishes a framework for investigating shared representations between cardiac electrical and acoustic signals, with the longer-term goal of developing multimodal representations that can transfer knowledge across modalities and with the longer-term goal of supporting robust cardiac AI in data-limited clinical environments. Ongoing work extends this retrieval framework with contrastive pretraining objectives and evaluates whether cross-modal embeddings transfer to downstream diagnostic tasks such as murmur and arrhythmia classification.