RADXAI: Evidence Grounded and Verifiable Reasoning for Medical VLMs Using Radiology Data

Vision Language Models (VLMs) have extensive clinical applications like automated report generation and medical Visual Question Answering (VQA) resulting in an increasing demand for medical vision language models. Despite their impressive fluency, these models can generate clinical findings that are unsupported by the input image, posing a critical barrier to trustworthy deployment. These hallucinations are poorly captured by conventional text generation metrics, which primarily assess lexical similarity rather than whether generated statements are grounded in visual evidence. This motivates us towards the need for a framework that can mitigate hallucinations in medical VLMs, and explicitly probe and quantify their hallucination behavior. In this work, we introduce RadXAI, a hallucination aware framework that combines Visual Retrieval Augmented Generation (VRAG) with Reinforcement Learning with Verifiable Rewards (RLVR) to improve the reliability and allow visual grounding of Medical VLMs. Here, we construct a clinically grounded, Visual Question Answering (VQA) dataset for chest X-rays from CheXpert Plus [1] to produce a report rq for any given query radiograph, xq. The pipeline does clinical entity extraction from raw radiology reports followed by deterministic VQA pair construction from the extracted entities. We adopt LLaVA-1.5-7B as the backbone and fine-tune it using clinical chest X-ray reports from the CheXpert Plus dataset. The model is finetuned on raw-reports for the frontal chest X-ray image of the patients followed by VRAG. VRAG augments the model with relevant reference cases retrieved from a medical image report repository, allowing the model to ground its predictions in visually and clinically related evidence rather than relying solely on parametric knowledge. The RLVR introduces verifiable reward signals that encourage the model to develop more reliable reasoning chains and favor clinically supported responses. To directly examine hallucination behavior, we further introduce a controlled probing protocol that evaluates the model under three settings: no retrieval, relevant VRAG retrieval, and a random-retrieval control. This design allows us to distinguish whether improvements arise simply from providing additional information or specifically from retrieving relevant evidence. Experiments are conducted across CheXpert Plus, MIMIC-CXR, and IU X-Ray datasets. The automatically constructed VQA test pairs for all the three radiology datasets are evaluated under three retrieval conditions. V-RAG fine tuning with retrieval raises finding level precision from 0.549 to 0.844 and Matthews Correlation Coefficient (MCC) from +0.159 to +0.472, corresponding to a reduction in the hallucination rate from 45.1% to 15.6%. The ordering no-retrieval < random < VRAG holds for both trained models on CheXpert Plus, MIMIC-CXR and IU X-Ray datasets. We found that retrieval relevance based on visual grounding drives hallucination reduction. To reduce the bias, we further do reasoning chains using verifiable rewards enabling the model to do better reasoning to provide reliable and fair answers