Timezone: »

 
Poster
Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharing
Mikhail Khodak · Renbo Tu · Tian Li · Liam Li · Maria-Florina Balcan · Virginia Smith · Ameet Talwalkar

Thu Dec 09 08:30 AM -- 10:00 AM (PST) @
in Poster Session 6 »

Tuning hyperparameters is a crucial but arduous part of the machine learning pipeline. Hyperparameter optimization is even more challenging in federated learning, where models are learned over a distributed network of heterogeneous devices; here, the need to keep data on device and perform local training makes it difficult to efficiently train and evaluate configurations. In this work, we investigate the problem of federated hyperparameter tuning. We first identify key challenges and show how standard approaches may be adapted to form baselines for the federated setting. Then, by making a novel connection to the neural architecture search technique of weight-sharing, we introduce a new method, FedEx, to accelerate federated hyperparameter tuning that is applicable to widely-used federated optimization methods such as FedAvg and recent variants. Theoretically, we show that a FedEx variant correctly tunes the on-device learning rate in the setting of online convex optimization across devices. Empirically, we show that FedEx can outperform natural baselines for federated hyperparameter tuning by several percentage points on the Shakespeare, FEMNIST, and CIFAR-10 benchmarks—obtaining higher accuracy using the same training budget.

Keywords: Deep Learning · Optimization · Machine Learning · Meta Learning · Federated Learning

Author Information

Mikhail Khodak (CMU)
Renbo Tu (CMU, Carnegie Mellon University)
Tian Li (CMU)
Liam Li (Carnegie Mellon University)
Maria-Florina Balcan (Carnegie Mellon University)
Virginia Smith (Carnegie Mellon University)
Ameet Talwalkar (CMU)

More from the Same Authors