Multi-Modal Conditional GAN: Data Synthesis in the Medical Domain

Despite continuous collection of linked clinical and imaging datasets within the drug development process it remains challenging to analyze those data to improve our understanding of disease and treatment. Data collection is often implemented inconsistently across studies or study sites, specific data modalities may be missing (e.g. lab measurements or medical images), and patient consent and data privacy laws constrain the purpose for which data may be used. In this paper we propose a method for conditional data generation across tabular and imaging modalities as a solution to overcome some of these challenges by generating synthetic patient data that are both realistic and complete across modalities. Our method, the multi-modal conditional GAN (MMCGAN), combines a conditional GAN for tabular data alongside a model for conditional 3D image synthesis at variable resolution. Our method brings a novel combination of capabilities: joint, scalable and efficient conditional data synthesis for clinical and full resolution 3D imaging data.