Position: A Minimum Reporting Specification for Regulatory Use of Medical-AI Red-Team Results
Abstract
Red teaming can expose failures in generative artificial intelligence (AI) used in healthcare, but a reported failure rate is meaningful only in relation to the system and evaluation protocol that produced it. We argue that adversarial-testing requirements should include a minimum reporting specification before results inform regulatory decisions or cross-system comparisons. A structured, non-exhaustive map of 25 purposively selected research sources and 13 governance documents, supplemented by targeted complementary evidence, motivates five interpretation problems: regression despite aggregate improvement; averages that conceal scenario-level failures or scoring effects; separation of content fidelity from behavioural safety; protocol-dependent multi-turn trajectories; and recommendation changes despite unchanged clinical facts. Mitigation can improve these outcomes, making matched retests as important as initial failure discovery. We propose Reporting of Red Teaming for Medical AI (RT-MED), a 12-item candidate extension to existing reporting guidance. Its contribution is to link clinical use and consequences with the target, attacker and judge configuration, attack opportunity, outcome denominator, and mitigation/retest record. These protocol-specific outcomes do not estimate deployed patient harm. RT-MED requires stakeholder consensus and independent validation before adoption as a reporting standard.