ISA-Bench: A Benchmark for Computational Reasoning Across Instruction Set Architectures

Can large language models write correct programs in unfamiliar, constrained instruction sets? We introduce ISA-Bench, a benchmark of 92 tasks across five programming game environments, each built around a distinct instruction set architecture. The tasks test computational reasoning primitives including algebraic derivation from minimal operations, sequential planning under register constraints, parallel decomposition, compositional hardware design, and spatial pattern generation. Each game includes a parser, execution engine, and test-case verifier for automated evaluation with structured feedback. We evaluate 14 open-weight models (8B–32B) under single-shot and iterative settings. Even the best model solves only half the tasks, and the dominant failure mode is producing syntactically invalid programs before execution begins. Iterative feedback with execution results recovers many failures caused by unfamiliar syntax, but provides no benefit when the model lacks the computational strategy a task requires, suggesting that current models cannot learn novel algorithmic reasoning from error signals alone.