PULSE: Surprise-Based Online Unsupervised Keyframe Detection

Keyframing provides a compact temporal abstraction for long-horizon robot manipulation by retaining only task-relevant transitions from otherwise redundant trajectories. We present PULSE, an online and unsupervised keyframe detector that identifies manipulation boundaries from predictive surprise without boundary supervision or future observations. PULSE learns visual and proprioceptive dynamics independently using two causal predictive branches. Visual surprise is measured as local novelty in a compact forecast-trained latent space, while proprioceptive surprise is measured from future-state prediction error. Each modality is robustly calibrated and thresholded independently, and only the resulting peak lists are fused, allowing either branch to detect events that are salient in one modality but weak in the other. We evaluate PULSE on manually annotated frame-level boundaries from RoboMME, RMBench, and MIKASA-Robo, covering single-arm, bimanual, visually driven, contact-rich, and memory-dependent manipulation. The fused detector achieves 0.719, 0.681, and 0.749 F1 at $\pm10$ frames, outperforming every evaluated baseline on all three benchmarks and the average of the top three baselines by 15.4%, 15.4%, and 22.5%, respectively. Task-level analysis further shows that visual surprise is especially effective for environment-driven events with weak kinematic signatures, whereas proprioceptive surprise dominates on contact- and robot-state-driven transitions. These results show that modality-specific predictive surprise provides an effective causal signal for unsupervised manipulation keyframe detection.