Timezone: »

 
GIST: Distributed Training for Large-Scale Graph Convolutional Networks
Cameron Wolfe · Jingkang Yang · Fangshuo Liao · Arindam Chowdhury · Chen Dun · Artun Bayer · Santiago Segarra · Anastasios Kyrillidis

@
in New Frontiers in Graph Learning »
Event URL: https://openreview.net/forum?id=-XsiFMpdSJz »
The graph convolutional network (GCN) is a go-to solution for machine learning on graphs, but its training is notoriously difficult to scale both in terms of graph size and the number of model parameters. Although some work has explored training on large-scale graphs, we pioneer efficient training of large-scale GCN models with the proposal of a novel, distributed training framework, called \texttt{GIST}. \texttt{GIST} disjointly partitions the parameters of a GCN model into several, smaller sub-GCNs that are trained independently and in parallel. Compatible with all GCN architectures and existing sampling techniques, \texttt{GIST} $i)$ improves model performance, $ii)$ scales to training on arbitrarily large graphs, $iii)$ decreases wall-clock training time, and $iv)$ enables the training of markedly overparameterized GCN models. Remarkably, with \texttt{GIST}, we train an astonishgly-wide $32,\!768$-dimensional GraphSAGE model, which exceeds the capacity of a single GPU by a factor of $8\times$, to SOTA performance on the Amazon2M dataset.
Keywords: graph neural networks · Distributed Training · overparameterization · Efficient Training

Author Information

Cameron Wolfe (Rice University)
Jingkang Yang (Nanyang Technological University)
Fangshuo Liao (Rice University)
Arindam Chowdhury (Rice University)
Chen Dun (Rice University)
Artun Bayer (Rice University)
Santiago Segarra (Rice University)
Anastasios Kyrillidis (Rice University)

More from the Same Authors