Beyond the Emulator: Bridging the Sim-to-Real Gap in Mobile Agent Benchmarking

Autonomous mobile agents powered by multimodal foundation models have demonstrated strong capabilities in task planning and UI navigation. However, existing benchmarks evaluate agents almost exclusively in synthetic desktop emulators, masking the physical constraints that govern real-world deployment on resource-constrained devices: battery drain, thermal throttling, dynamic power regimes, and end-to-end latency over wireless networks. To bridge this gap, we introduce BeYond The Emulator (BYTE), an open benchmarking framework that connects physical Android smartphones directly to foundation models using the Model Context Protocol (MCP). BYTE separates agent perception and action from out-of-band task evaluation, while a lightweight on-device service collects real-time battery, power, and thermal metrics on mobile devices over wireless ADB. Evaluating both cloud-offloaded and on-device foundation models on physical hardware uncovers deployment constraints that synthetic emulators fundamentally cannot capture: dynamic power envelopes, sustained thermal accumulation under inference, and non-linear energy scaling across task complexity and compute localities. These findings demonstrate that grounding evaluation in physical device telemetry is essential for assessing foundation model readiness under real-world resource constraints.