SynTeX-FL: Cross-Modal Text Transfer in Federated Learning for Medical Visual Question Answering

Medical visual question answering (VQA) and federated learning (FL) have gained prominence as critical methods for clinical artificial intelligence, with VQA supporting image-based diagnostic reasoning and FL allowing institutions to train models jointly without sharing sensitive patient data. However, existing approaches face substantial challenges in cross-modal vertical FL settings, where each participating institution possesses unimodal, modality-specific textual data without access to paired multimodal samples. To address this limitation, we introduce SynTeX-FL, a federated framework designed to enable cross-modal text transfer for medical VQA without sharing raw data. Moreover, SynTeX-FL applies a reconstruction-based text synthesis model that extracts clinically relevant semantics from medical images to generate cross-modal textual representations. Furthermore, the proposed method incorporates modality-specialized low-rank adaptation (LoRA) modules to enhance modality-aware representations for each imaging domain. During aggregation, the central server employs discriminator-based quality scores and source-aware weighting strategies to integrate heterogeneous client contributions. Extensive experiments on multimodal medical VQA benchmarks demonstrate that SynTeX-FL significantly outperforms current FL baselines, providing a robust and efficient solution for cross-modal reasoning in decentralized clinical environments.