Engaging Clinical Organisations in Language Model Method Development: Stakeholder Groups, Communication Practices, and Implications for Project Design

Hospitals assign a small set of administrative diagnostic codes to each emergency visit, and those codes, rather than a clinician's notes, form the datasets used for planning and research. We used a locally deployed large language model to read clinical notes and label each visit for alcohol, drug and self-harm involvement, recovering cases administrative codes missed and providing novel sub-domain characterisation of attendances. The purpose was to improve data quality and enlarge the feature space, allowing modelling, operational intelligence and pathway design to draw on variables coded data do not provide. Following development, expansion to partner emergency care organisations was identified as a critical step for the provision of actionable clinical information. The obstacles faced in achieving this were not primarily technical. Prospective partners raised reputational exposure, discomfort releasing confidential records for method development regardless of the recipient data environment, and uncertainty about whether the resulting measures would serve them or be used against them. This paper reports the researcher-perceived stakeholder groups that emerged in a publicly funded national health system, none fully anticipated at the outset, and the reframing each required before engagement could proceed. The unifying underlying difficulty is that health organisations have historically been measured almost exclusively as an instrument of accountability; any new measurement risks being viewed through that lens. Communication must establish which of two domains outputs belong to: measurement for accountability, and measurement to serve the organisation's own priorities. This paper reports safeguards, communication practices, two failures, and what to consider evidence of success.