Timezone: »
The representation learning on textual graph is to generate low-dimensional embeddings for the nodes based on the individual textual features and the neighbourhood information. Recent breakthroughs on pretrained language models and graph neural networks push forward the development of corresponding techniques. The existing works mainly rely on the cascaded model architecture: the textual features of nodes are independently encoded by language models at first; the textual embeddings are aggregated by graph neural networks afterwards. However, the above architecture is limited due to the independent modeling of textual features. In this work, we propose GraphFormers, where layerwise GNN components are nested alongside the transformer blocks of language models. With the proposed architecture, the text encoding and the graph aggregation are fused into an iterative workflow, making each node's semantic accurately comprehended from the global perspective. In addition, a progressive learning strategy is introduced, where the model is successively trained on manipulated data and original data to reinforce its capability of integrating information on graph. Extensive evaluations are conducted on three large-scale benchmark datasets, where GraphFormers outperform the SOTA baselines with comparable running efficiency. The source code is released at https://github.com/microsoft/GraphFormers .
Author Information
Junhan Yang (University of Science and Technology of China)
Zheng Liu (Microsoft)
Shitao Xiao (beijing university of posts and telecommunications)
Chaozhuo Li (Microsoft Research Asia)
Defu Lian (University of Science and Technology of China)
Sanjay Agrawal (Microsoft)
Amit Singh (Microsoft)
Guangzhong Sun (University of Science and Technology of China)
Xing Xie (Microsoft Research Asia)
More from the Same Authors
-
2023 Poster: Cross-links Matter for Link Prediction: Rethinking the Debiased GNN from a Data Perspective »
Zihan Luo · Jianxun Lian · Hong Huang · Xiran Song · Xing Xie · Hai Jin -
2023 Poster: Bayesian Active Causal Discovery with Multi-Fidelity Experiments »
Zeyu Zhang · Chaozhuo Li · Xu Chen · Xing Xie -
2023 Poster: Frequency-domain MLPs are More Effective Learners in Time Series Forecasting »
Kun Yi · Qi Zhang · Wei Fan · Hui He · Pengyang Wang · Shoujin Wang · Ning An · Defu Lian · Longbing Cao · Zhendong Niu -
2023 Poster: Knowledge Distillation for High Dimensional Search Index »
Zepu Lu · Jin Chen · Defu Lian · ZAIXI ZHANG · Yong Ge · Enhong Chen -
2023 Poster: Train Once and Explain Everywhere: Pre-training Interpretable Graph Neural Networks »
Jun Yin · Senzhang Wang · Hao Yan · Chaozhuo Li · Jianxun Lian -
2023 Poster: Model-enhanced Vector Index »
Hailin Zhang · Yujing Wang · Qi Chen · Ruiheng Chang · Ting Zhang · Ziming Miao · Yingyan Hou · Yang Ding · Xupeng Miao · Haonan Wang · Bochen Pang · Yuefeng Zhan · Hao Sun · Weiwei Deng · Qi Zhang · Fan Yang · Xing Xie · Mao Yang · Bin CUI -
2023 Poster: V-InFoR: A Robust Graph Neural Networks Explainer for Structurally Corrupted Graphs »
Jun Yin · Senzhang Wang · Chaozhuo Li · Xing Xie · Jianxin Wang -
2023 Poster: A Comprehensive Study on Text-attributed Graphs: Benchmarking and Rethinking »
Hao Yan · Chaozhuo Li · Ruosong Long · Chao Yan · Jianan Zhao · Wenwen Zhuang · Jun Yin · Peiyan Zhang · Weihao Han · Hao Sun · Weiwei Deng · Qi Zhang · Lichao Sun · Xing Xie · Senzhang Wang -
2022 Poster: A Neural Corpus Indexer for Document Retrieval »
Yujing Wang · Yingyan Hou · Haonan Wang · Ziming Miao · Shibin Wu · Hao Sun · Qi Chen · Yuqing Xia · Chengmin Chi · Guoshuai Zhao · Zheng Liu · Xing Xie · Hao Sun · Weiwei Deng · Qi Zhang · Mao Yang -
2022 Poster: USB: A Unified Semi-supervised Learning Benchmark for Classification »
Yidong Wang · Hao Chen · Yue Fan · Wang SUN · Ran Tao · Wenxin Hou · Renjie Wang · Linyi Yang · Zhi Zhou · Lan-Zhe Guo · Heli Qi · Zhen Wu · Yu-Feng Li · Satoshi Nakamura · Wei Ye · Marios Savvides · Bhiksha Raj · Takahiro Shinozaki · Bernt Schiele · Jindong Wang · Xing Xie · Yue Zhang -
2022 Poster: FairVFL: A Fair Vertical Federated Learning Framework with Contrastive Adversarial Learning »
Tao Qi · Fangzhao Wu · Chuhan Wu · Lingjuan Lyu · Tong Xu · Hao Liao · Zhongliang Yang · Yongfeng Huang · Xing Xie -
2022 Poster: Graph Convolution Network based Recommender Systems: Learning Guarantee and Item Mixture Powered Strategy »
Leyan Deng · Defu Lian · Chenwang Wu · Enhong Chen -
2022 Poster: Cache-Augmented Inbatch Importance Resampling for Training Recommender Retriever »
Jin Chen · Defu Lian · Yucheng Li · Baoyun Wang · Kai Zheng · Enhong Chen -
2022 Poster: Self-explaining deep models with logic rule reasoning »
Seungeon Lee · Xiting Wang · Sungwon Han · Xiaoyuan Yi · Xing Xie · Meeyoung Cha -
2022 Poster: Recommender Forest for Efficient Retrieval »
Chao Feng · Wuchao Li · Defu Lian · Zheng Liu · Enhong Chen -
2021 Poster: Meta-learning with an Adaptive Task Scheduler »
Huaxiu Yao · Yu Wang · Ying Wei · Peilin Zhao · Mehrdad Mahdavi · Defu Lian · Chelsea Finn -
2020 Poster: Sampling-Decomposable Generative Adversarial Recommender »
Binbin Jin · Defu Lian · Zheng Liu · Qi Liu · Jianhui Ma · Xing Xie · Enhong Chen