Timezone: »

 
Poster
Auto-Encoding Knowledge Graph for Unsupervised Medical Report Generation
Fenglin Liu · Chenyu You · Xian Wu · Shen Ge · Sheng wang · Xu Sun

Thu Dec 09 12:30 AM -- 02:00 AM (PST) @
in Poster Session 5 »

Medical report generation, which aims to automatically generate a long and coherent report of a given medical image, has been receiving growing research interests. Existing approaches mainly adopt a supervised manner and heavily rely on coupled image-report pairs. However, in the medical domain, building a large-scale image-report paired dataset is both time-consuming and expensive. To relax the dependency on paired data, we propose an unsupervised model Knowledge Graph Auto-Encoder (KGAE) which accepts independent sets of images and reports in training. KGAE consists of a pre-constructed knowledge graph, a knowledge-driven encoder and a knowledge-driven decoder. The knowledge graph works as the shared latent space to bridge the visual and textual domains; The knowledge-driven encoder projects medical images and reports to the corresponding coordinates in this latent space and the knowledge-driven decoder generates a medical report given a coordinate in this space. Since the knowledge-driven encoder and decoder can be trained with independent sets of images and reports, KGAE is unsupervised. The experiments show that the unsupervised KGAE generates desirable medical reports without using any image-report training pairs. Moreover, KGAE can also work in both semi-supervised and supervised settings, and accept paired images and reports in training. By further fine-tuning with image-report pairs, KGAE consistently outperforms the current state-of-the-art models on two datasets.

Keywords: Self-Supervised Learning · Graph Learning

Author Information

Fenglin Liu (Peking University)
Chenyu You (Yale University)

Chenyu You is a Ph.D. student in the Department of Electrical Engineering, at Yale University, working with Professor James Duncan. He obtained his master degree in Electrical Engineering from Stanford University, specializing in Artificial Intelligence (AI) Prior to that, he received his bachelor degree (with highest honors) in Electrical Engineering and Mathematics from Rensselaer Polytechnic Institute (RPI). He is broadly interested in the area of machine learning theory and algorithms intersecting the fields of computer & medical vision, natural language processing, signal processing, and distributed systems.

Xian Wu (Tencent)
Shen Ge (Tencent Medical AI Lab)
Sheng wang (University of Washington)
Xu Sun (Peking University)

More from the Same Authors