Timezone: »
Recent advances in neural architecture search inspire many channel number search algorithms~(CNS) for convolutional neural networks. To improve searching efficiency, parameter sharing is widely applied, which reuses parameters among different channel configurations. Nevertheless, it is unclear how parameter sharing affects the searching process. In this paper, we aim at providing a better understanding and exploitation of parameter sharing for CNS. Specifically, we propose affine parameter sharing~(APS) as a general formulation to unify and quantitatively analyze existing channel search algorithms. It is found that with parameter sharing, weight updates of one architecture can simultaneously benefit other candidates. However, it also results in less confidence in choosing good architectures. We thus propose a new strategy of parameter sharing towards a better balance between training efficiency and architecture discrimination. Extensive analysis and experiments demonstrate the superiority of the proposed strategy in channel configuration against many state-of-the-art counterparts on benchmark datasets.
Author Information
Jiaxing Wang (Institute of Automation, Chinese Academy of Sciences)
Haoli Bai (The Chinese University of Hong Kong)
Jiaxiang Wu (Tencent AI Lab)
Xupeng Shi (Northeastern University)
Junzhou Huang (University of Texas at Arlington / Tencent AI Lab)
Irwin King (Chinese University of Hong Kong)
Michael R Lyu (CUHK)
Jian Cheng (Institute of Automation, Chinese Academy of Sciences)
More from the Same Authors
-
2021 : Score-based Graph Generative Model for Neutrino Events Classification and Reconstruction »
Yiming Sun · Zixing Song · Irwin King -
2022 Poster: GLIF: A Unified Gated Leaky Integrate-and-Fire Neuron for Spiking Neural Networks »
Xingting Yao · Fanrong Li · Zitao Mo · Jian Cheng -
2022 Poster: PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficient »
Weihan Cao · Yifan Zhang · Jianfei Gao · Anda Cheng · Ke Cheng · Jian Cheng -
2022 Poster: SCL-WC: Cross-Slide Contrastive Learning for Weakly-Supervised Whole-Slide Image Classification »
Xiyue Wang · Jinxi Xiang · Jun Zhang · Sen Yang · Zhongyi Yang · Ming-Hui Wang · Jing Zhang · Wei Yang · Junzhou Huang · Xiao Han -
2022 : Fast and Accurate Antibody Structure Prediction without Sequence Homologs »
Jiaxiang Wu · Fandi Wu · Biaobin Jiang · Wei Liu · Peilin Zhao -
2022 : Individual Fairness in Dynamic Financial Networks »
Zixing Song · Yueen Ma · Irwin King -
2023 Poster: Optimal Block-wise Asymmetric Graph Construction for Graph-based Semi-supervised Learning »
Zixing Song · Yifei Zhang · Irwin King -
2023 Poster: Predicting Global Label Relationship Matrix for Graph Neural Networks under Heterophily »
Langzhang Liang · Xiangjing Hu · Zenglin Xu · Zixing Song · Irwin King -
2023 Poster: Towards Efficient and Accurate Winograd Convolution via Full Quantization »
Chen Tianqi · Weixiang Xu · Weihan Chen · Peisong Wang · Jian Cheng -
2023 Poster: Mitigating the Popularity Bias in Graph-based Collaborative Filtering »
Yifei Zhang · Hao Zhu · yankai Chen · Zixing Song · Piotr Koniusz · Irwin King -
2023 Poster: Automatic Grouping for Efficient Cooperative Multi-Agent Reinforcement Learning »
Yifan Zang · Jinmin He · Kai Li · Haobo Fu · Qiang Fu · Junliang Xing · Jian Cheng -
2023 Poster: Retaining Beneficial Information from Detrimental Data for Neural Network Repair »
Long-Kai Huang · Peilin Zhao · Junzhou Huang · Sinno Pan -
2023 Poster: No Change, No Gain: Empowering Graph Neural Networks with Expected Model Change Maximization for Active Learning »
Zixing Song · Yifei Zhang · Irwin King -
2022 Spotlight: PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficient »
Weihan Cao · Yifan Zhang · Jianfei Gao · Anda Cheng · Ke Cheng · Jian Cheng -
2022 Spotlight: Lightning Talks 2B-2 »
Chenjian Gao · Rui Ding · Lingzhi LI · Fan Yang · Xingting Yao · Jianxin Li · Bing Su · Zhen Shen · Tongda Xu · Shuai Zhang · Ji-Rong Wen · Lin Guo · Fanrong Li · Kehua Guo · Zhongshu Wang · Zhi Chen · Xiangyuan Zhu · Zitao Mo · Dailan He · Hui Xiong · Yan Wang · Zheng Wu · Wenbing Tao · Jian Cheng · Haoyi Zhou · Li Shen · Ping Tan · Liwei Wang · Hongwei Qin -
2022 Spotlight: GLIF: A Unified Gated Leaky Integrate-and-Fire Neuron for Spiking Neural Networks »
Xingting Yao · Fanrong Li · Zitao Mo · Jian Cheng -
2022 Poster: Towards Efficient Post-training Quantization of Pre-trained Language Models »
Haoli Bai · Lu Hou · Lifeng Shang · Xin Jiang · Irwin King · Michael R Lyu -
2022 Poster: Singular Value Fine-tuning: Few-shot Segmentation requires Few-parameters Fine-tuning »
Yanpeng Sun · Qiang Chen · Xiangyu He · Jian Wang · Haocheng Feng · Junyu Han · Errui Ding · Jian Cheng · Zechao Li · Jingdong Wang -
2021 Poster: Functionally Regionalized Knowledge Transfer for Low-resource Drug Discovery »
Huaxiu Yao · Ying Wei · Long-Kai Huang · Ding Xue · Junzhou Huang · Zhenhui (Jessie) Li -
2021 Poster: Not All Low-Pass Filters are Robust in Graph Convolutional Networks »
Heng Chang · Yu Rong · Tingyang Xu · Yatao Bian · Shiji Zhou · Xin Wang · Junzhou Huang · Wenwu Zhu -
2020 Poster: Dirichlet Graph Variational Autoencoder »
Jia Li · Jianwei Yu · Jiajin Li · Honglei Zhang · Kangfei Zhao · Yu Rong · Hong Cheng · Junzhou Huang -
2020 Poster: RetroXpert: Decompose Retrosynthesis Prediction Like A Chemist »
Chaochao Yan · Qianggang Ding · Peilin Zhao · Shuangjia Zheng · JINYU YANG · Yang Yu · Junzhou Huang -
2020 Poster: Unsupervised Text Generation by Learning from Search »
Jingjing Li · Zichao Li · Lili Mou · Xin Jiang · Michael R Lyu · Irwin King -
2020 Spotlight: RetroXpert: Decompose Retrosynthesis Prediction Like A Chemist »
Chaochao Yan · Qianggang Ding · Peilin Zhao · Shuangjia Zheng · JINYU YANG · Yang Yu · Junzhou Huang -
2020 Poster: Self-Supervised Graph Transformer on Large-Scale Molecular Data »
Yu Rong · Yatao Bian · Tingyang Xu · Weiyang Xie · Ying Wei · Wenbing Huang · Junzhou Huang -
2020 Poster: Deep Multimodal Fusion by Channel Exchanging »
Yikai Wang · Wenbing Huang · Fuchun Sun · Tingyang Xu · Yu Rong · Junzhou Huang -
2020 Poster: Adversarial Sparse Transformer for Time Series Forecasting »
Sifan Wu · Xi Xiao · Qianggang Ding · Peilin Zhao · Ying Wei · Junzhou Huang -
2019 : MicroNet Challenge »
Peisong Wang · Cong Leng · Jian Cheng · Zhongxia Yan · Hanrui Wang · Trevor Gale · Erich Elsen -
2019 Poster: Hyperparameter Learning via Distributional Transfer »
Ho Chung Law · Peilin Zhao · Leung Sing Chan · Junzhou Huang · Dino Sejdinovic -
2019 Poster: DTWNet: a Dynamic Time Warping Network »
Xingyu Cai · Tingyang Xu · Jinfeng Yi · Junzhou Huang · Sanguthevar Rajasekaran -
2019 Poster: NAT: Neural Architecture Transformer for Accurate and Compact Architectures »
Yong Guo · Yin Zheng · Mingkui Tan · Qi Chen · Jian Chen · Peilin Zhao · Junzhou Huang -
2019 Poster: Double Quantization for Communication-Efficient Distributed Optimization »
Yue Yu · Jiaxiang Wu · Longbo Huang -
2019 Poster: Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement »
Chao Yang · Xiaojian Ma · Wenbing Huang · Fuchun Sun · Huaping Liu · Junzhou Huang · Chuang Gan -
2019 Spotlight: Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement »
Chao Yang · Xiaojian Ma · Wenbing Huang · Fuchun Sun · Huaping Liu · Junzhou Huang · Chuang Gan -
2018 Poster: Discrimination-aware Channel Pruning for Deep Neural Networks »
Zhuangwei Zhuang · Mingkui Tan · Bohan Zhuang · Jing Liu · Yong Guo · Qingyao Wu · Junzhou Huang · Jinhui Zhu -
2018 Poster: Weakly Supervised Dense Event Captioning in Videos »
Xin Wang · Wenbing Huang · Chuang Gan · Jingdong Wang · Wenwu Zhu · Junzhou Huang -
2018 Poster: Almost Optimal Algorithms for Linear Stochastic Bandits with Heavy-Tailed Payoffs »
Han Shao · Xiaotian Yu · Irwin King · Michael R Lyu -
2018 Spotlight: Almost Optimal Algorithms for Linear Stochastic Bandits with Heavy-Tailed Payoffs »
Han Shao · Xiaotian Yu · Irwin King · Michael R Lyu -
2018 Poster: Adaptive Sampling Towards Fast Graph Representation Learning »
Wenbing Huang · Tong Zhang · Yu Rong · Junzhou Huang -
2017 Poster: Efficient Optimization for Linear Dynamical Systems with Applications to Clustering and Sparse Coding »
Wenbing Huang · Mehrtash Harandi · Tong Zhang · Lijie Fan · Fuchun Sun · Junzhou Huang -
2014 Poster: Combinatorial Pure Exploration of Multi-Armed Bandits »
Shouyuan Chen · Tian Lin · Irwin King · Michael R Lyu · Wei Chen -
2014 Oral: Combinatorial Pure Exploration of Multi-Armed Bandits »
Shouyuan Chen · Tian Lin · Irwin King · Michael R Lyu · Wei Chen -
2013 Poster: Exact and Stable Recovery of Pairwise Interaction Tensors »
Shouyuan Chen · Michael R Lyu · Irwin King · Zenglin Xu -
2013 Spotlight: Exact and Stable Recovery of Pairwise Interaction Tensors »
Shouyuan Chen · Michael R Lyu · Irwin King · Zenglin Xu -
2012 Poster: Compressive Sensing MRI with Wavelet Tree Sparsity »
Chen Chen · Junzhou Huang -
2010 Workshop: Machine Learning for Social Computing »
Zenglin Xu · Irwin King · Shenghuo Zhu · Yuan Qi · Rong Yan · John Yen -
2009 Poster: Adaptive Regularization for Transductive Support Vector Machine »
Zenglin Xu · Rong Jin · Jianke Zhu · Irwin King · Michael R Lyu · Zhirong Yang -
2009 Spotlight: Adaptive Regularization for Transductive Support Vector Machine »
Zenglin Xu · Rong Jin · Jianke Zhu · Irwin King · Michael R Lyu · Zhirong Yang -
2009 Poster: Heavy-Tailed Symmetric Stochastic Neighbor Embedding »
Zhirong Yang · Irwin King · Zenglin Xu · Erkki Oja -
2009 Spotlight: Heavy-Tailed Symmetric Stochastic Neighbor Embedding »
Zhirong Yang · Irwin King · Zenglin Xu · Erkki Oja -
2008 Poster: Learning with Consistency between Inductive Functions and Kernels »
Haixuan Yang · Irwin King · Michael R Lyu -
2008 Spotlight: Learning with Consistency between Inductive Functions and Kernels »
Haixuan Yang · Irwin King · Michael R Lyu -
2008 Poster: An Extended Level Method for Efficient Multiple Kernel Learning »
Zenglin Xu · Rong Jin · Irwin King · Michael R Lyu -
2007 Poster: Efficient Convex Relaxation for Transductive Support Vector Machine »
Zenglin Xu · Rong Jin · Jianke Zhu · Irwin King · Michael R Lyu