Timezone: »

 
Poster
Deep Graph Pose: a semi-supervised deep graphical model for improved animal pose tracking
Anqi Wu · Estefany Kelly Buchanan · Matthew Whiteway · Michael Schartner · Guido Meijer · Jean-Paul Noel · Erica Rodriguez · Claire Everett · Amy Norovich · Evan Schaffer · Neeli Mishra · C. Daniel Salzman · Dora Angelaki · Andrés Bendesky · The International Brain Laboratory The International Brain Laboratory · John Cunningham · Liam Paninski

Thu Dec 10 09:00 AM -- 11:00 AM (PST) @ Poster Session 5 #1575
in Poster Session 6 »

Noninvasive behavioral tracking of animals is crucial for many scientific investigations. Recent transfer learning approaches for behavioral tracking have considerably advanced the state of the art. Typically these methods treat each video frame and each object to be tracked independently. In this work, we improve on these methods (particularly in the regime of few training labels) by leveraging the rich spatiotemporal structures pervasive in behavioral video --- specifically, the spatial statistics imposed by physical constraints (e.g., paw to elbow distance), and the temporal statistics imposed by smoothness from frame to frame. We propose a probabilistic graphical model built on top of deep neural networks, Deep Graph Pose (DGP), to leverage these useful spatial and temporal constraints, and develop an efficient structured variational approach to perform inference in this model. The resulting semi-supervised model exploits both labeled and unlabeled frames to achieve significantly more accurate and robust tracking while requiring users to label fewer training frames. In turn, these tracking improvements enhance performance on downstream applications, including robust unsupervised segmentation of behavioral syllables,'' and estimation of interpretabledisentangled'' low-dimensional representations of the full behavioral video. Open source code is available at \href{\CodeLink}{https://github.com/paninski-lab/deepgraphpose}.

Author Information

Anqi Wu (Columbia University)
Estefany Kelly Buchanan (Columbia University)
Matthew Whiteway (Columbia University)
Michael Schartner (University of Geneva)
Guido Meijer (Champalimaud Center for the Unknown)
Jean-Paul Noel (New York University)
Erica Rodriguez (Columbia University)
Claire Everett (Columbia University)
Amy Norovich (Columbia University)
Evan Schaffer (Columbia University)
Neeli Mishra (Columbia University)
C. Daniel Salzman (Columbia University)
Dora Angelaki (New York University)
Andrés Bendesky (Columbia University)
The International Brain Laboratory The International Brain Laboratory (The International Brain Laboratory)
John Cunningham (University of Columbia)
Liam Paninski (Columbia University)

More from the Same Authors