Disagreement Without Difference: Auditing a Benchmark Cluster

The community has built many benchmarks for Islamic and Arabic cultural knowledge in large language models, and has not checked whether they measure the same thing. We evaluate a fixed panel of open models across the public benchmarks in this area and compare the model ranking each one produces. Taken together the cluster appears incoherent: two halves of the same benchmark rank models alike, but two different benchmarks do not. That summary is misleading. We introduce a three-part diagnostic screen, computed from each benchmark in isolation and never from cross-benchmark agreement, which splits the cluster cleanly. The instruments that pass it agree with one another about as closely as each agrees with itself; those that fail do not. The failure modes are concrete and mechanical: on one benchmark a majority of models never read the question and score exactly the base rate of one answer, and on another the gold answer occupies the first position on almost every item. Disagreement here is the work of a few identifiable instruments rather than a general unreliability, and a single inference pass is enough to find them.