Timezone: »
Recent advances in neural architecture search inspire many channel number search algorithms~(CNS) for convolutional neural networks. To improve searching efficiency, parameter sharing is widely applied, which reuses parameters among different channel configurations. Nevertheless, it is unclear how parameter sharing affects the searching process. In this paper, we aim at providing a better understanding and exploitation of parameter sharing for CNS. Specifically, we propose affine parameter sharing~(APS) as a general formulation to unify and quantitatively analyze existing channel search algorithms. It is found that with parameter sharing, weight updates of one architecture can simultaneously benefit other candidates. However, it also results in less confidence in choosing good architectures. We thus propose a new strategy of parameter sharing towards a better balance between training efficiency and architecture discrimination. Extensive analysis and experiments demonstrate the superiority of the proposed strategy in channel configuration against many state-of-the-art counterparts on benchmark datasets.
Author Information
Jiaxing Wang (Institute of Automation, Chinese Academy of Sciences)
Haoli Bai (The Chinese University of Hong Kong)
Jiaxiang Wu (Tencent AI Lab)
Xupeng Shi (Northeastern University)
Junzhou Huang (University of Texas at Arlington / Tencent AI Lab)
Irwin King (Chinese University of Hong Kong)
Michael Lyu (CUHK)
Jian Cheng (Institute of Automation, Chinese Academy of Sciences)
More from the Same Authors
-
2020 Poster: Dirichlet Graph Variational Autoencoder »
Jia Li · Jianwei Yu · Jiajin Li · Honglei Zhang · Kangfei Zhao · Yu Rong · Hong Cheng · Junzhou Huang -
2020 Poster: RetroXpert: Decompose Retrosynthesis Prediction Like A Chemist »
Chaochao Yan · Qianggang Ding · Peilin Zhao · Shuangjia Zheng · JINYU YANG · Yang Yu · Junzhou Huang -
2020 Poster: Unsupervised Text Generation by Learning from Search »
Jingjing Li · Zichao Li · Lili Mou · Xin Jiang · Michael Lyu · Irwin King -
2020 Spotlight: RetroXpert: Decompose Retrosynthesis Prediction Like A Chemist »
Chaochao Yan · Qianggang Ding · Peilin Zhao · Shuangjia Zheng · JINYU YANG · Yang Yu · Junzhou Huang -
2020 Poster: Self-Supervised Graph Transformer on Large-Scale Molecular Data »
Yu Rong · Yatao Bian · Tingyang Xu · Weiyang Xie · Ying WEI · Wenbing Huang · Junzhou Huang -
2020 Poster: Deep Multimodal Fusion by Channel Exchanging »
Yikai Wang · Wenbing Huang · Fuchun Sun · Tingyang Xu · Yu Rong · Junzhou Huang -
2020 Poster: Adversarial Sparse Transformer for Time Series Forecasting »
Sifan Wu · Xi Xiao · Qianggang Ding · Peilin Zhao · Ying Wei · Junzhou Huang -
2019 Poster: Hyperparameter Learning via Distributional Transfer »
Ho Chung Law · Peilin Zhao · Leung Sing Chan · Junzhou Huang · Dino Sejdinovic -
2019 Poster: DTWNet: a Dynamic Time Warping Network »
Xingyu Cai · Tingyang Xu · Jinfeng Yi · Junzhou Huang · Sanguthevar Rajasekaran -
2019 Poster: NAT: Neural Architecture Transformer for Accurate and Compact Architectures »
Yong Guo · Yin Zheng · Mingkui Tan · Qi Chen · Jian Chen · Peilin Zhao · Junzhou Huang -
2019 Poster: Double Quantization for Communication-Efficient Distributed Optimization »
Yue Yu · Jiaxiang Wu · Longbo Huang -
2019 Poster: Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement »
Chao Yang · Xiaojian Ma · Wenbing Huang · Fuchun Sun · Huaping Liu · Junzhou Huang · Chuang Gan -
2019 Spotlight: Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement »
Chao Yang · Xiaojian Ma · Wenbing Huang · Fuchun Sun · Huaping Liu · Junzhou Huang · Chuang Gan -
2018 Poster: Discrimination-aware Channel Pruning for Deep Neural Networks »
Zhuangwei Zhuang · Mingkui Tan · Bohan Zhuang · Jing Liu · Yong Guo · Qingyao Wu · Junzhou Huang · Jinhui Zhu -
2018 Poster: Weakly Supervised Dense Event Captioning in Videos »
Xin Wang · Wenbing Huang · Chuang Gan · Jingdong Wang · Wenwu Zhu · Junzhou Huang -
2018 Poster: Almost Optimal Algorithms for Linear Stochastic Bandits with Heavy-Tailed Payoffs »
Han Shao · Xiaotian Yu · Irwin King · Michael Lyu -
2018 Spotlight: Almost Optimal Algorithms for Linear Stochastic Bandits with Heavy-Tailed Payoffs »
Han Shao · Xiaotian Yu · Irwin King · Michael Lyu -
2018 Poster: Adaptive Sampling Towards Fast Graph Representation Learning »
Wenbing Huang · Tong Zhang · Yu Rong · Junzhou Huang -
2017 Poster: Efficient Optimization for Linear Dynamical Systems with Applications to Clustering and Sparse Coding »
Wenbing Huang · Mehrtash Harandi · Tong Zhang · Lijie Fan · Fuchun Sun · Junzhou Huang -
2014 Poster: Combinatorial Pure Exploration of Multi-Armed Bandits »
Shouyuan Chen · Tian Lin · Irwin King · Michael Lyu · Wei Chen -
2014 Oral: Combinatorial Pure Exploration of Multi-Armed Bandits »
Shouyuan Chen · Tian Lin · Irwin King · Michael Lyu · Wei Chen -
2013 Poster: Exact and Stable Recovery of Pairwise Interaction Tensors »
Shouyuan Chen · Michael Lyu · Irwin King · Zenglin Xu -
2013 Spotlight: Exact and Stable Recovery of Pairwise Interaction Tensors »
Shouyuan Chen · Michael Lyu · Irwin King · Zenglin Xu -
2012 Poster: Compressive Sensing MRI with Wavelet Tree Sparsity »
Chen Chen · Junzhou Huang -
2010 Workshop: Machine Learning for Social Computing »
Zenglin Xu · Irwin King · Shenghuo Zhu · Yuan Qi · Rong Yan · John Yen -
2009 Poster: Adaptive Regularization for Transductive Support Vector Machine »
Zenglin Xu · Rong Jin · Jianke Zhu · Irwin King · Michael Lyu · Zhirong Yang -
2009 Spotlight: Adaptive Regularization for Transductive Support Vector Machine »
Zenglin Xu · Rong Jin · Jianke Zhu · Irwin King · Michael Lyu · Zhirong Yang -
2009 Poster: Heavy-Tailed Symmetric Stochastic Neighbor Embedding »
Zhirong Yang · Irwin King · Zenglin Xu · Erkki Oja -
2009 Spotlight: Heavy-Tailed Symmetric Stochastic Neighbor Embedding »
Zhirong Yang · Irwin King · Zenglin Xu · Erkki Oja -
2008 Poster: Learning with Consistency between Inductive Functions and Kernels »
Haixuan Yang · Irwin King · Michael Lyu -
2008 Spotlight: Learning with Consistency between Inductive Functions and Kernels »
Haixuan Yang · Irwin King · Michael Lyu -
2008 Poster: An Extended Level Method for Efficient Multiple Kernel Learning »
Zenglin Xu · Rong Jin · Irwin King · Michael Lyu -
2007 Poster: Efficient Convex Relaxation for Transductive Support Vector Machine »
Zenglin Xu · Rong Jin · Jianke Zhu · Irwin King · Michael Lyu