Timezone: »

Stochastic Optimization with Laggard Data Pipelines
Naman Agarwal · Rohan Anil · Tomer Koren · Kunal Talwar · Cyril Zhang

Wed Dec 09 09:00 AM -- 11:00 AM (PST) @ Poster Session 3 #1054

State-of-the-art optimization is steadily shifting towards massively parallel pipelines with extremely large batch sizes. As a consequence, CPU-bound preprocessing and disk/memory/network operations have emerged as new performance bottlenecks, as opposed to hardware-accelerated gradient computations. In this regime, a recently proposed approach is data echoing (Choi et al., 2019), which takes repeated gradient steps on the same batch while waiting for fresh data to arrive from upstream. We provide the first convergence analyses of "data-echoed" extensions of common optimization methods, showing that they exhibit provable improvements over their synchronous counterparts. Specifically, we show that in convex optimization with stochastic minibatches, data echoing affords speedups on the curvature-dominated part of the convergence rate, while maintaining the optimal statistical rate.

Author Information

Naman Agarwal (Google)
Rohan Anil (Google)
Tomer Koren (Tel Aviv University & Google)
Kunal Talwar (Apple)
Cyril Zhang (Microsoft Research NYC)

More from the Same Authors