Timezone: »

Hyperparameter Transfer Across Developer Adjustments
Danny Stoll

After developer adjustments to a machine learning (ML) system, how can the results of an old hyperparameter optimization automatically be used to speed up a new hyperparameter optimization? This question poses a challenging problem, as developer adjustments can change which hyperparameter settings perform well, or even the hyperparameter space itself. While many approaches exist that leverage knowledge obtained on previous tasks, so far, knowledge from previous development steps remains entirely untapped. In this work, we remedy this situation and propose a new research framework: hyperparameter transfer across adjustments (HT-AA). To lay a solid foundation for this research framework, we provide four HT-AA baseline algorithms and eight benchmarks. The best baseline, on average, reaches a given performance 2x faster than a prominent HPO algorithm without transfer. As hyperparameter optimization is a crucial step in ML development but requires extensive computational resources, this speed up would lead to faster development cycles, lower costs, and reduced environmental impacts. To make these benefits available to ML developers off-the-shelf, we provide a python package that implements the proposed transfer algorithm.

Author Information

Danny Stoll (University of Freiburg)
Danny Stoll

First doctoral candidate at the engineering faculty of the University of Freiburg admitted directly after the B.Sc. Working on AutoML, Deep Learning, Neural Architecture Search, Hyperparameter Optimization, Joint Architecture and Hyperparameter Search, Meta-Learning

More from the Same Authors