Timezone: »

Chest ImaGenome Dataset for Clinical Reasoning
Joy T Wu · Nkechinyere Agu · Ismini Lourentzou · Arjun Sharma · Joseph Alexander Paguio · Jasper Seth Yao · Edward C Dee · William Mitchell · Satyananda Kashyap · Andrea Giovannini · Leo Anthony Celi · Mehdi Moradi

Tue Dec 07 01:35 AM -- 01:45 AM (PST) @

Despite the progress in automatic detection of radiologic findings from chest X-ray (CXR) images in recent years, a quantitative evaluation of the explainability of these models is hampered by the lack of locally labeled datasets for different findings. With the exception of a few expert-labeled small-scale datasets for specific findings, such as pneumonia and pneumothorax, most of the CXR deep learning models to date are trained on global "weak" labels extracted from text reports, or trained via a joint image and unstructured text learning strategy. Inspired by the Visual Genome effort in the computer vision community, we constructed the first Chest ImaGenome dataset with a scene graph data structure to describe 242,072 images. Local annotations are automatically produced using a joint rule-based natural language processing (NLP) and atlas-based bounding box detection pipeline. Through a radiologist constructed CXR ontology, the annotations for each CXR are connected as an anatomy-centered scene graph, useful for image-level reasoning and multimodal fusion applications. Overall, we provide: i) 1,256 combinations of relation annotations between 29 CXR anatomical locations (objects with bounding box coordinates) and their attributes, structured as a scene graph per image, ii) over 670,000 localized comparison relations (for improved, worsened, or no change) between the anatomical locations across sequential exams, as well as ii) a manually annotated gold standard scene graph dataset from 500 unique patients.

Author Information

Joy T Wu (Almaden Research Center, International Business Machines)
Nkechinyere Agu (Rensselaer Polytechnic Institute)
Ismini Lourentzou (University of Illinois at Urbana Champaign)
Arjun Sharma (International Business Machines)
Joseph Alexander Paguio (Albert Einstein Healthcare Network)
Jasper Seth Yao (Albert Einstein Healthcare Network-Philadelphia Campus)
Edward C Dee (Harvard Medical School)
William Mitchell (Harvard University)
Satyananda Kashyap (IBM Research)

Satyananda Kashyap's research focuses on developing novel machine learning techniques to tackle problems in healthcare. Before joining IBM Research, he pursued his Ph.D. at the University of Iowa. At Iowa, his work focused on developing novel graph-based machine learning algorithms on longitudinal Knee MRIs to quantify and understand the degeneration of the knee joint with the progression of osteoarthritis. Currently, his research focuses on the problem of chest x-ray specifically on developing explainable AI methods for classifying the various diseases so that the machine diagnosis can be understood by a human and trusted.

Andrea Giovannini (Swiss Federal Institute of Technology)
Leo Anthony Celi (Massachusetts Institute of Technology)
Mehdi Moradi (IBM Research)

More from the Same Authors