Timezone: »
While other areas of machine learning have seen more and more automation, designing a high-performing recommender system still requires a high level of human effort. Furthermore, recent work has shown that modern recommender system algorithms do not always improve over well-tuned baselines. A natural follow-up question is, "how do we choose the right algorithm for a new dataset and performance metric?" In this work, we start by giving the first large-scale study of recommender system approaches by comparing 24 algorithms and 100 sets of hyperparameters across 85 datasets and 315 metrics. We find that the best algorithms and hyperparameters are highly dependent on the dataset and performance metric. However, there is also a strong correlation between the performance of each algorithm and various meta-features of the datasets. Motivated by these findings, we create RecZilla, a meta-learning approach to recommender systems that uses a model to predict the best algorithm and hyperparameters for new, unseen datasets. By using far more meta-training data than prior work, RecZilla is able to substantially reduce the level of human involvement when faced with a new recommender system application. We not only release our code and pretrained RecZilla models, but also all of our raw experimental results, so that practitioners can train a RecZilla model for their desired performance metric: https://github.com/naszilla/reczilla.
Author Information
Duncan McElfresh (Stanford University)
Sujay Khandagale (Columbia University)
Jonathan Valverde (University of Maryland, College Park)
John Dickerson (Arthur AI & University of Maryland)
Colin White (Abacus.AI)
More from the Same Authors
-
2021 : Synthetic Benchmarks for Scientific Research in Explainable Machine Learning »
Yang Liu · Sujay Khandagale · Colin White · Willie Neiswanger -
2021 : Learning Revenue-Maximizing Auctions With Differentiable Matching »
Michael Curry · Uro Lyi · Tom Goldstein · John P Dickerson -
2021 : Learning Revenue-Maximizing Auctions With Differentiable Matching »
Michael Curry · Uro Lyi · Tom Goldstein · John P Dickerson -
2021 : An mHealth Intervention for African American and Hispanic Adults: Preliminary Results from a One-Year Field Test »
Christine Herlihy · John Dickerson -
2021 : An mHealth Intervention for African American and Hispanic Adults: Preliminary Results from a One-Year Field Test »
Christine Herlihy · John Dickerson -
2022 : A Deep Dive into Dataset Imbalance and Bias in Face Identification »
Valeriia Cherepanova · Steven Reich · Samuel Dooley · Hossein Souri · John Dickerson · Micah Goldblum · Tom Goldstein -
2022 : Tensions Between the Proxies of Human Values in AI »
Daniel Nissani · Teresa Datta · John Dickerson · Max Cembalest · Akash Khanna · Haley Massa -
2022 : AutoML for Climate Change: A Call to Action »
Renbo Tu · Nicholas Roberts · Vishak Prasad C · Sibasis Nayak · Paarth Jain · Frederic Sala · Ganesh Ramakrishnan · Ameet Talwalkar · Willie Neiswanger · Colin White -
2022 : Characterizing Anomalies with Explainable Classifiers »
Naveen Durvasula · Valentine d Hauteville · Keegan Hines · John Dickerson -
2022 : A Deep Dive into Dataset Imbalance and Bias in Face Identification »
Valeriia Cherepanova · Steven Reich · Samuel Dooley · Hossein Souri · John Dickerson · Micah Goldblum · Tom Goldstein -
2022 : On the Importance of Architectures and Hyperparameters for Fairness in Face Recognition »
Samuel Dooley · Rhea Sukthanker · John Dickerson · Colin White · Frank Hutter · Micah Goldblum -
2022 : On the Importance of Architectures and Hyperparameters for Fairness in Face Recognition »
Samuel Dooley · Rhea Sukthanker · John Dickerson · Colin White · Frank Hutter · Micah Goldblum -
2022 : A Deep Dive into Dataset Imbalance and Bias in Face Identification »
Valeriia Cherepanova · Steven Reich · Samuel Dooley · Hossein Souri · John Dickerson · Micah Goldblum · Tom Goldstein -
2023 Poster: Doubly Constrained Fair Clustering »
John Dickerson · Seyed Esmaeili · Jamie Morgenstern · Claire Jie Zhang -
2023 Poster: Fair, Polylog-Approximate Low-Cost Hierarchical Clustering »
Marina Knittel · Max Springer · John Dickerson · MohammadTaghi Hajiaghayi -
2023 Poster: Diffused Redundancy in Pre-trained Representations »
Vedant Nanda · Till Speicher · John Dickerson · Krishna Gummadi · Soheil Feizi · Adrian Weller -
2023 Poster: Reward Scale Robustness for Proximal Policy Optimization via DreamerV3 Tricks »
Ryan Sullivan · Akarsh Kumar · Shengyi Huang · John Dickerson · Joseph Suarez -
2023 Poster: Rethinking Bias Mitigation: Fairer Architectures Make for Fairer Face Recognition »
Samuel Dooley · Rhea Sukthanker · John Dickerson · Colin White · Frank Hutter · Micah Goldblum -
2022 Workshop: Graph Learning for Industrial Applications: Finance, Crime Detection, Medicine and Social Media »
Manuela Veloso · John Dickerson · Senthil Kumar · Eren K. · Jian Tang · Jie Chen · Peter Henstock · Susan Tibbs · Ani Calinescu · Naftali Cohen · C. Bayan Bruss · Armineh Nourbakhsh -
2022 Social: Open Mic Night »
John Dickerson -
2022 Poster: Robustness Disparities in Face Detection »
Samuel Dooley · George Z Wei · Tom Goldstein · John Dickerson -
2022 Poster: NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies »
Arjun Krishnakumar · Colin White · Arber Zela · Renbo Tu · Mahmoud Safari · Frank Hutter -
2021 Poster: How Powerful are Performance Predictors in Neural Architecture Search? »
Colin White · Arber Zela · Robin Ru · Yang Liu · Frank Hutter -
2021 Poster: VQ-GNN: A Universal Framework to Scale up Graph Neural Networks using Vector Quantization »
Mucong Ding · Kezhi Kong · Jingling Li · Chen Zhu · John Dickerson · Furong Huang · Tom Goldstein -
2021 Poster: Fair Clustering Under a Bounded Cost »
Seyed Esmaeili · Brian Brubach · Aravind Srinivasan · John Dickerson -
2021 Poster: PreferenceNet: Encoding Human Preferences in Auction Design with Deep Learning »
Neehar Peri · Michael Curry · Samuel Dooley · John Dickerson -
2021 Poster: NAS-Bench-x11 and the Power of Learning Curves »
Shen Yan · Colin White · Yash Savani · Frank Hutter -
2021 Poster: How does a Neural Network's Architecture Impact its Robustness to Noisy Labels? »
Jingling Li · Mozhi Zhang · Keyulu Xu · John Dickerson · Jimmy Ba -
2020 Workshop: Workshop on Dataset Curation and Security »
Nathalie Baracaldo · Yonatan Bisk · Avrim Blum · Michael Curry · John Dickerson · Micah Goldblum · Tom Goldstein · Bo Li · Avi Schwarzschild -
2020 Poster: A Study on Encodings for Neural Architecture Search »
Colin White · Willie Neiswanger · Sam Nolen · Yash Savani -
2020 Spotlight: A Study on Encodings for Neural Architecture Search »
Colin White · Willie Neiswanger · Sam Nolen · Yash Savani -
2020 Poster: Detection as Regression: Certified Object Detection with Median Smoothing »
Ping-yeh Chiang · Michael Curry · Ahmed Abdelkader · Aounon Kumar · John Dickerson · Tom Goldstein -
2020 Poster: Intra-Processing Methods for Debiasing Neural Networks »
Yash Savani · Colin White · Naveen Sundar Govindarajulu -
2020 Poster: Certifying Strategyproof Auction Networks »
Michael Curry · Ping-yeh Chiang · Tom Goldstein · John Dickerson -
2020 Poster: Improving Policy-Constrained Kidney Exchange via Pre-Screening »
Duncan McElfresh · Michael Curry · Tuomas Sandholm · John Dickerson -
2020 Poster: Probabilistic Fair Clustering »
Seyed Esmaeili · Brian Brubach · Leonidas Tsepenekas · John Dickerson -
2019 : Coffee/Poster session 1 »
Shiro Takagi · Khurram Javed · Johanna Sommer · Amr Sharaf · Pierluca D'Oro · Ying Wei · Sivan Doveh · Colin White · Santiago Gonzalez · Cuong Nguyen · Mao Li · Tianhe Yu · Tiago Ramalho · Masahiro Nomura · Ahsan Alvi · Jean-Francois Ton · W. Ronny Huang · Jessica Lee · Sebastian Flennerhag · Michael Zhang · Abram Friesen · Paul Blomstedt · Alina Dubatovka · Sergey Bartunov · Subin Yi · Iaroslav Shcherbatyi · Christian Simon · Zeyuan Shang · David MacLeod · Lu Liu · Liam Fowl · Diego Mesquita · Deirdre Quillen -
2019 Poster: Making the Cut: A Bandit-based Approach to Tiered Interviewing »
Candice Schumann · Zhi Lang · Jeffrey Foster · John Dickerson -
2019 Poster: Adversarial training for free! »
Ali Shafahi · Mahyar Najibi · Mohammad Amin Ghiasi · Zheng Xu · John Dickerson · Christoph Studer · Larry Davis · Gavin Taylor · Tom Goldstein -
2018 Poster: Data-Driven Clustering via Parameterized Lloyd's Families »
Maria-Florina Balcan · Travis Dick · Colin White -
2018 Spotlight: Data-Driven Clustering via Parameterized Lloyd's Families »
Maria-Florina Balcan · Travis Dick · Colin White -
2015 : Uncertainty in Dynamic Matching »
John P Dickerson