Still Citing the Old Law? Statutory Knowledge and Temporal Anchoring in Thai Legal Question Answering
Abstract
People who cannot afford a lawyer increasingly ask language models about the law. We ask whether those models know Thai statutory law, and whether they keep answering with rules that have since been repealed. We build a bilingual benchmark of 60 synthetic vignettes covering four dated amendments to Thai law between eight months and five years old. Fourteen of the vignettes test rules that did not change over that period. We evaluate nine models across two prompt languages with three runs each (3,036 responses). The clearest result is not about staleness. On the unchanged rules the most recent frontier models answer 93-100% correctly while a Thai-specific model answers 31-41% and a Chinese open model 64-66%: a gap of roughly sixty percentage points, though the two weaker models are also the smaller ones, so training and capability are not separated here. Anchoring to repealed rules is a real but smaller effect, concentrated in older model versions, and it declines in five of the six old/new model pairs we test. Two hypotheses we set out to test do not hold: anchoring does not fade monotonically with amendment age, and Thai prompts are not uniformly worse than English. Across all wrong answers, 97.7% contain no expression of uncertainty.