1 in 4 Committed Links, 1 in 34 Mentions: When Aggregate Metrics Hide Canonicalization Failures in Clinical Knowledge Graphs
Rui Ding ⋅ Shira M Lupkin ⋅ Nicole Tebaldi
Abstract
Clinical NLP pipelines routinely map an extracted entity mention onto a controlled vocabulary by taking its nearest neighbor in an embedding space. The step has no way to say "no matching concept exists"; a similarity cutoff is the only brake, and the cutoff cannot separate a correct match from a confident wrong one. We measure how often the links it does commit are wrong. On BC5CDR, where gold concept identifiers are available, one in four rewrites at the platform-default cutoff points to a different concept than the mention, and one in every $34$ mentions in the corpus is silently mis-linked. Raising the cutoff does not solve this; it only trades away coverage. The error rate never falls below $9\%$ at any threshold we report, and stays above $12\%$ for all three embedders once coverage is held equal. An expert adjudicated every rewrite committed on six independently built clinical knowledge graphs and found the same rate ($22.5\%$ of $120$); an LLM-panel replication on the ACI-Bench dialogue corpus found a higher one ($35.5\%$). Individual mistakes recur verbatim across both corpora and across seven separately built extraction pipelines, which points at the shared embedding and vocabulary and not at any one dataset. Downstream scores can miss all of this: adding a second canonicalization layer at the platform-default cutoff leaves question answering over these graphs unchanged, while at a lower cutoff the same layer degrades all five graphs we tested. We therefore treat the problem as one of provenance, and report two deployment patterns, together called NAC (Negation-Aware Canonicalization): keep the originally extracted mention on every rewritten node, so a wrong link can still be found and audited, and block rewrites that flip a mention's negation status. Although we study clinical knowledge graphs, the same failure mode applies wherever nearest-neighbor linking runs against an incomplete vocabulary and cannot abstain.
Chat is not available.
Successful Page Load