FemLex: Model Failure or Representation Gap? Auditing Women’s Health Language in Clinical AI

Clinical NLP systems are typically evaluated on how accurately they map patient language to standardized medical concepts, but what happens when the target representation itself cannot adequately encode the input? We investigate this overlooked source of error using FemLex, an empirical audit of 3,361 women’s health narratives grounded by clinician annotation and evaluated across generative and retrieval-based models. We find that representational inadequacy is both substantial and difficult to infer from conventional model outputs: 47.5% of the clinician reference was unmapped to SNOMED-CT or ICD-11, while BioLORD embedding similarity substantially overlapped across exact, partial, and unmapped representations, yielding a three-way macro-F1 of 0.362. Model behavior also diverged markedly on clinician-unmapped narratives, and we identify high-similarity force-fit cases where retrieval produced plausible concepts despite loss of the intended meaning. Together, these results expose a hidden assumption in clinical NLP evaluation: retrieval quality and representational adequacy are not equivalent. FemLex motivates evaluating whether a model recovers an appropriate clinical concept, and also how model performance should be interpreted when the available target representation is insufficient.