Scene Physics Acquisition via Visual De-animation

We introduce a new paradigm for fast and rich physical scene understanding without human annotations. At the core of our system is a physical world representation recovered by a perception module and utilized by physics and graphics engines. During training, the perception module and the generative models learn by visual de-animation --- interpreting and reconstructing the visual information stream. During testing, the system first recovers the physical world state, and then uses the generative models for reasoning and future prediction. Unlike forward simulation, inverting a physics or graphics engine is a computationally hard problem; we overcome this challenge through the use of a convolutional inversion network. Our system quickly recognizes the physical world state from appearance and motion cues, and has the flexibility to incorporate both differentiable and non-differentiable physics and graphics engines. We evaluate our system on both synthetic and real datasets involving multiple physical scenes, and demonstrate that our system performs well on both physical state estimation and reasoning problems. We further show that the knowledge learned on the synthetic dataset generalizes to constrained real images.