MotorSense: A Video-EMG Dataset of Motoric Representations for Action Understanding

Computer vision models learn hand action only from its visual consequences, without access to the underlying embodied motor signals. We address this absence with MotorSense - the first large-scale paired egocentric video and surface electromyography (EMG) dataset. MotorSense pairs bimanual EMG bands with Meta Aria glasses and yields synchronized video, bimanual EMG and IMU, and audio across $25$ hours of unscripted manipulation. Beyond scale, EMG offers an alternative to manual annotation, and captures finger movement, hand configuration, and forces exerted between hand and environment --- this is in contrast to manual annotation which is costly, subjective, and unreliable under occlusion. EMG provides a dense hand signal free of both occlusion and labeler bias. To bridge the representational gap between low-level EMG and high-level semantic action representations, we learn a mid-level representation over the EMG data that we call motor codes. We utilize these motor codes in two settings: 1) motor-code pre-training, in which image and video backbones are pre-trained to predict motor codes from pixels, and subsequently evaluated over downstream tasks of 3D hand-object reconstruction and action recognition; 2) motor-code conditioning, in which the codes are used as control inputs for a video diffusion model, yielding videos of fine-grained hand manipulation. On action recognition (EPIC-Kitchens, Ego-Exo4D, and Meccano), under a frozen-evaluation protocol we observe a large $35.8$\% mean relative improvement in top-1 accuracy over conventional pre-trained backbones. On ARCTIC 3D hand-object reconstruction we observe a sizeable increase in object-pose success rate from $72.90$\% to $77.22$\% compared to the pre-training of the base ARCTIC model. Code and data are made anonymously available at https://anonymous.4open.science/r/MotorSense-7173/ .