Timezone: »

 
Poster
Rethinking and Scaling Up Graph Contrastive Learning: An Extremely Efficient Approach with Group Discrimination
YIZHEN ZHENG · Shirui Pan · Vincent CS Lee · Yu Zheng · Philip S Yu

Thu Dec 01 09:00 AM -- 11:00 AM (PST) @ Hall J #208
in Poster Session 5 »

Graph contrastive learning (GCL) alleviates the heavy reliance on label information for graph representation learning (GRL) via self-supervised learning schemes. The core idea is to learn by maximising mutual information for similar instances, which requires similarity computation between two node instances. However, GCL is inefficient in both time and memory consumption. In addition, GCL normally requires a large number of training epochs to be well-trained on large-scale datasets. Inspired by an observation of a technical defect (i.e., inappropriate usage of Sigmoid function) commonly used in two representative GCL works, DGI and MVGRL, we revisit GCL and introduce a new learning paradigm for self-supervised graph representation learning, namely, Group Discrimination (GD), and propose a novel GD-based method called Graph Group Discrimination (GGD). Instead of similarity computation, GGD directly discriminates two groups of node samples with a very simple binary cross-entropy loss. In addition, GGD requires much fewer training epochs to obtain competitive performance compared with GCL methods on large-scale datasets. These two advantages endow GGD with very efficient property. Extensive experiments show that GGD outperforms state-of-the-art self-supervised methods on eight datasets. In particular, GGD can be trained in 0.18 seconds (6.44 seconds including data preprocessing) on ogbn-arxiv, which is orders of magnitude (10,000+) faster than GCL baselines while consuming much less memory. Trained with 9 hours on ogbn-papers100M with billion edges, GGD outperforms its GCL counterparts in both accuracy and efficiency.

Keywords: Graph Representation Learning · Self-supervised Graph Representation Learning · Unsupervised Graph Representation Learning · Graph Contrastive Learning

Author Information

YIZHEN ZHENG
Shirui Pan (Griffith University)
Vincent CS Lee (Monash University)

Vincent CS Lee (PhD) is currently an Associate Professor at Machine learning and Deep Learning Discipline of the Department of Data Science and Artificial Intelligence, Faculty of IT, Monash University, Australia. He received Australia Federal Government scholarship to pursue PhD from 1988 through to 1991 at The University of New Castle, NSW, in Australia. In 1973 to 1974, he was awarded a joint research scholarship by Ministry of Defence (Singapore) and Ministry of Defence (UK) for postgraduate study at Royal Air Force College, UK in aircraft electrical and instrument systems. He was a visiting academic to Tsinghua University in Beijing at the School of Economics and Management from Nov 2006 to March 2007; and was also a Visiting Professor to Information Communication Institute of Singapore from July 1994 through June 1995. Lee has 35 (13 years with MNC and public sector establishments in Singapore, and 22 years with four universities in Australia and Singapore) years of experience in applied and fundamental research, business enterprise system development, engineering and ICT system integration, and university teaching. Lee is also a registered professional electrical engineer in Singapore; Licensed Electrical Engineer (High voltage switching authorisation) in Singapore; Chartered Engineer, Senior members of IEEE USA. Lee published 200+ peer-reviewed international (high impact factor SCI and SSCI) Q1 journals articles and A and A* peerreviewed international conference proceedings. His publications appear in IEEE Transactions on Knowledge and Data Engineering; IEE Proceeding on Generation, Transmission and Utilisation; IEEE Transactions on Neural Networks and linear Systems, IEEE Transactions on Signal Processing; IEEE Selected Areas in Communications; IEEE Security and Privacy; European Journal of Operational Research; Applied Mathematics and Computation; Neuro-computing; Accounting and Finance; Journal of Wealth Management; and in AAAI, ICDE, ICWS, ICIS proceedings. He has chaired/co-chaired more than a dozen of IEEE international conference technical program committees: conference chair KICSS2012; general chair ICDIM2011; co-chair workshop on Intelligent Environment (IE2010); conference chair ICSSSM2008, and special sessions of IEEE International Conference on Service Computing, senior program committee members of IJCAI2020 Fintech special track, PC members of IJCNN2019, PAKDD2020, invited co-chair and speaker IEEE ICET2021, Beijing, etc. Lee has been awarded 22 internationally competitive research grants in signal processing, decision support systems, AI/ML, Digital health informatics, and financial engineering research projects. Lee's current research interests are multidisciplinary spreading across signal processing; adaptive knowledge representation and information engineering; data, text, and graph mining for knowledge discovery; decision theory; information system research based on design science paradigm; and Neuro-financial engineering, ICU patient health support systems, Thyroid cancer prognostic and treatment analytics. He has attracted research grants from ARC DP and LPs, A*STAR, and Swift Institute, UK. Dr Lee is the sole recipient of Monash University, Dean of IT, 2016 excellence Award for Higher Degree supervision. He has todate supervised to the award of 19 PhDs and 35 Research Masters students in signal processing, power systems quality, e-commerce, financial engineering, Digital healthcare support systems, Secured digital networking, and innovation management from commencement to the final award of thesis during his employ with Nanyang Technological University in Singapore, Swinburne University of Technology and Monash University both in Melbourne, Australia. Currently he is the principal supervisor of 7 PhDs in Machine Learning and Artificial Intelligence, ICU patient vital sign health monitoring, extrem learning system for Thyroid cancer patient diagnostic and treatment quality; and cosupervisors for 6 PhDs in deep learning for health modelling of smart infrastructure transport network, multiparty computation using cryptography techniques, and educational data mining research.

Yu Zheng (Latrobe University)
Philip S Yu (UIC)

More from the Same Authors