Where Does Reasoning Fail? Controlled Analogy Evaluation for Lower-Resource Languages

Language models often score lower on reasoning tasks outside English, but the reason is unclear because the instruction, question, script, and requested explanation often change together. We begin from a concrete Hindi result. On 405 analogy questions, three multilingual models with 8B to 9B parameters reached their best scores with English task framing, while structured step-by-step prompting added only 0.27 accuracy points on average. Yet the analogy and answer options remained in Hindi. The result therefore shows a framing effect, but it does not reveal whether the difficulty comes from the question language or the model’s reasoning process. We use this finding to design a controlled analogy benchmark for lower-resource languages, starting with Hindi. The study has three experiments. The first separates instruction, question, and explanation language. The second tests the effect of Hindi script. The third provides word meanings, supporting facts, or relation information to locate where errors occur. This work-in-progress paper presents the motivating evidence, benchmark design, experimental comparisons, and planned evaluation.