Timezone: »
The calculation of gradients and other forms of derivatives is a core part of machine learning, computer vision, and physical simulation. But the manual creation of derivatives is prone to error and requires a high "mental overhead" for practitioners in these fields. However, the process of taking derivatives is actually the highly mechanical application of the chain rule and can be computed using formal techniques such as automatic or symbolic differentiation. A family of "autodiff" approaches exist, each with their own particular strengths and tradeoffs.
In the ideal case, automatically generated derivatives should be competitive with manually generated ones and run at near-peak performance on modern hardware, but the most expressive systems for autodiff which can handle arbitrary, Turing-complete programs, are unsuited for performance-critical applications, such as large-scale machine learning or physical simulation. Alternatively, the most performant systems are not designed for use outside of their designated application space, e.g. graphics or neural networks. This workshop will bring together developers and researchers of state-of-the-art solutions to generating derivatives automatically and discuss ways in which these solutions can be evolved to be both more expressive and achieve higher performance. Topics for discussion will include:
- Whether it is feasible to create a single differentiable programming language, or if we will always have separate solutions for different fields such as vision and ML.
- What are the primitive data types of a differentiable language? N-dimensional arrays are useful for many machine learning applications, but other domains make use of graph types and sparse matrices.
- What are the challenges in elevating an expressive autodiff implementation from just a “prototyping language” to one used directly in performance-critical industrial settings?
- A shared representation of programs like LLVM IR has transformed programming language and compiler research. Is there any benefit to a common representation of differentiable programs that would enable shared tooling amongst autodiff libraries and implementations?
Sat 12:00 a.m. - 12:10 a.m.
|
Opening Remarks
(
Intro
)
|
🔗 |
Sat 12:10 a.m. - 12:40 a.m.
|
Barak A. Pearlmutter – Automatic Differentiation: History and Headroom
(
Talk
)
|
Barak A. Pearlmutter 🔗 |
Sat 12:50 a.m. - 1:20 a.m.
|
David Duvenaud – No more mini-languages: The power of autodiffing full-featured Python
(
Talk
)
|
David Duvenaud 🔗 |
Sat 1:30 a.m. - 2:00 a.m.
|
Coffee Break 1
|
🔗 |
Sat 2:00 a.m. - 2:30 a.m.
|
Jeff Dean – TensorFlow: Future Directions for Simplifying Large-Scale Machine Learning
(
Talk
)
|
Jeff Dean 🔗 |
Sat 2:40 a.m. - 5:30 a.m.
|
Lunch break
|
🔗 |
Sat 5:30 a.m. - 6:00 a.m.
|
Yoshua Bengio – Credit assignment: beyond backpropagation
(
Talk
)
|
Yoshua Bengio 🔗 |
Sat 6:10 a.m. - 6:40 a.m.
|
Matthew Johnson – Autodiff writes your exponential family inference code
(
Talk
)
|
Matthew Johnson 🔗 |
Sat 6:50 a.m. - 7:30 a.m.
|
Coffee Break 2
|
🔗 |
Sat 7:30 a.m. - 8:00 a.m.
|
Jeffrey M. Siskind – The tension between convenience and performance in automatic differentiation
(
Talk
)
|
Jeffrey Siskind 🔗 |
Sat 8:10 a.m. - 9:10 a.m.
|
Panel Discussion
(
Panel
)
|
🔗 |
Author Information
Alex Wiltschko (Twitter)
Zachary DeVito (Stanford)
Frederic Bastien (MILA - University of Montreal)
Pascal Lamblin (Google)
More from the Same Authors
-
2018 Poster: Automatic differentiation in ML: Where we are and where we should be going »
Bart van Merriënboer · Olivier Breuleux · Arnaud Bergeron · Pascal Lamblin -
2018 Spotlight: Automatic differentiation in ML: Where we are and where we should be going »
Bart van Merriënboer · Olivier Breuleux · Arnaud Bergeron · Pascal Lamblin -
2017 Workshop: The future of gradient-based machine learning software & techniques »
Alex Wiltschko · Bart van Merriënboer · Pascal Lamblin -
2016 Poster: Composing graphical models with neural networks for structured representations and fast inference »
Matthew Johnson · David Duvenaud · Alex Wiltschko · Ryan Adams · Sandeep R Datta -
2007 Poster: Learning the 2-D Topology of Images »
Nicolas Le Roux · Yoshua Bengio · Pascal Lamblin · Marc Joliveau · Balázs Kégl -
2006 Poster: Greedy Layer-Wise Training of Deep Networks »
Yoshua Bengio · Pascal Lamblin · Dan Popovici · Hugo Larochelle -
2006 Talk: Greedy Layer-Wise Training of Deep Networks »
Yoshua Bengio · Pascal Lamblin · Dan Popovici · Hugo Larochelle