Benchmark to Audit LLM Generated Clinical Notes for Disparities Arising from Biases and Stereotypes

After each patient encounter, physicians compile extensive, semi-structured clinical summaries known as SOAP notes. These notes, while essential for both clinical practice and research, are time-consuming to generate in a digital format, contributing significantly to physician burnout. Recently, large language models (LLMs) have shown promising abilities in automating the generation of SOAP notes. Despite these advancements, there is a risk that such models could inadvertently cause harm and worsen existing health disparities. It is crucial to systematically evaluate model failures related to equity to ensure the development of clinical documentation tools that uphold principles of health equity. This study introduces a benchmark dataset and proposes methodologies for assessing equity-related harms in LLM-generated, long-form SOAP notes. Our work aims to establish a foundation for ensuring that automated clinical documentation tools are not only efficient but also equitable in their impact on diverse patient populations.