Timezone: »
Learning representations of neural network weights given a model zoo is an emerg- ing and challenging area with many potential applications from model inspection, to neural architecture search or knowledge distillation. Recently, an autoencoder trained on a model zoo was able to learn a hyper-representation, which captures intrinsic and extrinsic properties of the models in the zoo. In this work, we ex- tend hyper-representations for generative use to sample new model weights. We propose layer-wise loss normalization which we demonstrate is key to generate high-performing models and several sampling methods based on the topology of hyper-representations. The models generated using our methods are diverse, per- formant and capable to outperform strong baselines as evaluated on several down- stream tasks: initialization, ensemble sampling and transfer learning. Our results indicate the potential of knowledge aggregation from model zoos to new models via hyper-representations thereby paving the avenue for novel research directions.
Author Information
Konstantin Schürholt (University of St. Gallen)
Boris Knyazev (University of Guelph)
Xavier Giro-i-Nieto (UPC Barcelona)
Xavier Giro-i-Nieto is an associate professor at the Universitat Politecnica de Catalunya (UPC). He graduated in Electrical Engineering studies at ETSETB (UPC) in 2000, after completing his master thesis on image compression at the Vrije Universiteit in Brussels (VUB) under the direction of Professor Peter Schelkens. In 2001 he worked in the digital television group of Sony Brussels, before returning to Barcelona and joining the Image Processing Group at the UPC. Since 2003, he has created and taught graduate and undergraduate courses for Electrical Engineering degress at the ESEIAAT and ETSETB schools from UPC. In 2013 he participated in the design of the Master in Computer Vision of Barcelona by UPC, UAB, UPF and UOC universities, where he lectures on deep learning, image retrieval and video processingl. He has taught several international courses in the framework of the European Erasmus program. He obtained his Phd on image retrieval in 2012, under the supervision by Professor Ferran Marqués from UPC and Professor Shih-Fu Chang from Columbia University. He was a visiting scholar during Summers 2008 to 2014 at the Digital Video and MultiMedia laboratory at Columbia University, in New York. His relation with industry includes collaborations with Mediapro, Catalan Broadcast Corporation (TV3), Pixable, Catchoom and Narrative.
Damian Borth (University of St.Gallen (HSG))
Related Events (a corresponding poster, oral, or spotlight)
-
2022 Poster: Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights »
Thu. Dec 1st 05:00 -- 07:00 PM Room Hall J #605
More from the Same Authors
-
2022 : Federated Continual Learning to Detect Accounting Anomalies in Financial Auditing »
Marco Schreyer · Hamed Hemati · Damian Borth · Miklos Vasarhelyi -
2022 : Towards dynamical stability analysis of sustainable power grids using Graph Neural Networks »
Christian Nauck · Michael Lindner · Ulrich Schürholt · Frank Hellmann -
2022 Spotlight: Lightning Talks 2A-3 »
David Buterez · Chengan He · Xuan Kan · Yutong Lin · Konstantin Schürholt · Yu Yang · Louis Annabi · Wei Dai · Xiaotian Cheng · Alexandre Pitti · Ze Liu · Jon Paul Janet · Jun Saito · Boris Knyazev · Mathias Quoy · Zheng Zhang · James Zachary · Steven J Kiddle · Xavier Giro-i-Nieto · Chang Liu · Hejie Cui · Zilong Zhang · Hakan Bilen · Damian Borth · Dino Oglic · Holly Rushmeier · Han Hu · Xiangyang Ji · Yi Zhou · Nanning Zheng · Ying Guo · Pietro Liò · Stephen Lin · Carl Yang · Yue Cao -
2022 Poster: Model Zoos: A Dataset of Diverse Populations of Neural Network Models »
Konstantin Schürholt · Diyar Taskiran · Boris Knyazev · Xavier Giró-i-Nieto · Damian Borth -
2021 Poster: Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Prediction »
Konstantin Schürholt · Dimche Kostadinov · Damian Borth -
2019 : Speech in Pixels: Automatic Detection of Offensive Memes for Moderation »
Xavier Giro-i-Nieto -
2018 : Contributed Work »
Thaer Moustafa Dieb · Aditya Balu · Amir H. Khasahmadi · Viraj Shah · Boris Knyazev · Payel Das · Garrett Goh · Georgy Derevyanko · Gianni De Fabritiis · Reiko Hagawa · John Ingraham · David Belanger · Jialin Song · Kim Nicoli · Miha Skalic · Michelle Wu · Niklas Gebauer · Peter Bjørn Jørgensen · Ryan-Rhys Griffiths · Shengchao Liu · Sheshera Mysore · Hai Leong Chieu · Philippe Schwaller · Bart Olsthoorn · Bianca-Cristina Cristescu · Wei-Cheng Tseng · Seongok Ryu · Iddo Drori · Kevin Yang · Soumya Sanyal · Zois Boukouvalas · Rishi Bedi · Arindam Paul · Sambuddha Ghosal · Daniil Bash · Clyde Fare · Zekun Ren · Ali Oskooei · Minn Xuan Wong · Paul Sinz · Théophile Gaudin · Wengong Jin · Paul Leu -
2017 : Coffee break and Poster Session I »
Nishith Khandwala · Steve Gallant · Gregory Way · Aniruddh Raghu · Li Shen · Aydan Gasimova · Alican Bozkurt · William Boag · Daniel Lopez-Martinez · Ulrich Bodenhofer · Samaneh Nasiri GhoshehBolagh · Michelle Guo · Christoph Kurz · Kirubin Pillay · Kimis Perros · George H Chen · Alexandre Yahi · Madhumita Sushil · Sanjay Purushotham · Elena Tutubalina · Tejpal Virdi · Marc-Andre Schulz · Samuel Weisenthal · Bharat Srikishan · Petar Veličković · Kartik Ahuja · Andrew Miller · Erin Craig · Disi Ji · Filip Dabek · Chloé Pou-Prom · Hejia Zhang · Janani Kalyanam · Wei-Hung Weng · Harish Bhat · Hugh Chen · Simon Kohl · Mingwu Gao · Tingting Zhu · Ming-Zher Poh · Iñigo Urteaga · Antoine Honoré · Alessandro De Palma · Maruan Al-Shedivat · Pranav Rajpurkar · Matthew McDermott · Vincent Chen · Yanan Sui · Yun-Geun Lee · Li-Fang Cheng · Chen Fang · Sibt ul Hussain · Cesare Furlanello · Zeev Waks · Hiba Chougrad · Hedvig Kjellstrom · Finale Doshi-Velez · Wolfgang Fruehwirt · Yanqing Zhang · Lily Hu · Junfang Chen · Sunho Park · Gatis Mikelsons · Jumana Dakka · Stephanie Hyland · yann chevaleyre · Hyunwoo Lee · Xavier Giro-i-Nieto · David Kale · Michael Hughes · Gabriel Erion · Rishab Mehra · William Zame · Stojan Trajanovski · Prithwish Chakraborty · Kelly Peterson · Muktabh Mayank Srivastava · Amy Jin · Heliodoro Tejeda Lemus · Priyadip Ray · Tamas Madl · Joseph Futoma · Enhao Gong · Syed Rameel Ahmad · Eric Lei · Ferdinand Legros