Translated, Not Transferred: Temporal Confidence Signals Across Languages and Local Knowledge
Abstract
Confidence-aware LLM systems are typically evaluated in English, leaving open whether they are reliable for the majority of the world that isn't. In this work, we separate two sources of unreliability that prior multilingual works often conflate, language shift and knowledge shift. Using a temporal framing of pre-solve Feeling-of-Knowing (FOK) and post-solve Judgement-of-Learning (JOL) LLM confidence inspired by meta-memory theory, we evaluate two frontier LLMs on English reasoning and factual benchmarks, their direct Marathi translations, and a genuinely local Marathi benchmark based on regional news sources. We find that confidence calibration survives translation largely intact but collapses on local knowledge. Notably, AUROC falls to near-random (0.52-0.56) even though the language is identical to the translated split. Post-solve confidence is generally, though not uniformly, better calibrated than the cheaper pre-solve signal, and can worsen after reasoning in some domains. Hallucinated answers keep persistently high confidence regardless of language, distinguishable by confidence level but not by how confidence changes during reasoning. Together, these results show that confidence calibration in low-resource settings requires further refinement and analysis before reliable deployment.