WorldMirror: Agentic Real-to-Sim Generation of Interactive Worlds from In-the-Wild Robot Demonstrations

Evaluating generalist robot policies at scale requires simulation environments that reproduce the observations and interactions of real deployments. Building such environments remains difficult: plausible assets alone do not ensure that objects, support surfaces, robots, and cameras are correctly aligned. We introduce WorldMirror, an agentic real-to-sim framework that constructs interactive digital twins from robot videos and recorded trajectories. A coding agent decomposes the scene, coordinates specialized reconstruction tools, and revises an executable world program using visual and simulation feedback. A layered representation separates static context from interactive assets, while camera-anchored assembly aligns the environment and robot for replay without trajectory modification. On DROID, WorldMirror improves perceptual fidelity and reproduces demonstrated motion and contact more faithfully than the evaluated baselines. For policy evaluation, we reconstruct all 18 RoboDojo-Real tasks and execute three policies trained on real-world data. Their aggregate simulated success rates closely track the real-world leaderboard, with Pearson r=0.9998 and MMRV=0.000. These results support agentic world construction as a practical approach to building simulation environments for robot policy evaluation.