Differentiable computer vision, graphics, and physics in machine learning

“Differentiable programs” are parameterized programs that allow themselves to be rewritten by gradient-based optimization. They are ubiquitous in modern-day machine learning. Recently, explicitly encoding our knowledge of the rules of the world in the form of differentiable programs has become more popular. In particular, differentiable realizations of well-studied processes such as physics, rendering, projective geometry, optimization to name a few, have enabled the design of several novel learning techniques. For example, many approaches have been proposed for unsupervised learning of depth estimation from unlabeled videos. Differentiable 3D reconstruction pipelines have demonstrated the potential for task-driven representation learning. A number of differentiable rendering approaches have been shown to enable single-view 3D reconstruction and other inverse graphics tasks (without requiring any form of 3D supervision). Differentiable physics simulators are being built to perform physical parameter estimation from video or for model-predictive control. While these advances have largely occurred in isolation, recent efforts have attempted to bridge the gap between the aforementioned areas. Narrowing the gaps between these otherwise isolated disciplines holds tremendous potential to yield new research directions and solve long-standing problems, particularly in understanding and reasoning about the 3D world.

Hence, we propose the “first workshop on differentiable computer vision, graphics, and physics in machine learning” with the aim of:
1. Narrowing the gap and fostering synergies between the computer vision, graphics, physics, and machine learning communities
2. Debating the promise and perils of differentiable methods, and identifying challenges that need to be overcome
3. Raising awareness about these techniques to the larger ML community
4. Discussing the broader impact of such techniques, and any ethical implications thereof.