Rank Collapse in Graph Attention Networks: Topology at Initialisation

Graphs provide a general language for relational structure, from molecules and recommender systems to social networks and physical systems. Graph neural networks (GNNs) learn node embeddings by combining information across a graph. Depth should let them capture long range structure, but deeper GNNs often become less informative as their embeddings collapse into a low dimensional space. Rank collapse removes independent directions and can appear as oversmoothing across nodes and feature overcorrelation across channels [1]. The resulting loss of node distinctions can harm tasks such as identifying different atomic roles in molecular property prediction. Crucially, attention does not remove this failure: even graph attention networks (GATs) can lose expressive power exponentially with depth [2]. At random initialisation, four main ingredients shape a GAT attention matrix: graph support and node degrees; node features; the initialised parameters that produce the scores; and softmax normalisation within each neighbourhood. Other architectural choices can also matter. Existing theory mostly studies collapse after repeated propagation. Recent work on dense transformers shows that rank collapse can begin at initialisation: a gap between the leading and remaining singular values can already push representations towards low rank [3]. We bring this view to graph attention, where topology and the score model jointly determine the initial attention matrix. This creates a question: how do graph topology and attention score statistics shape the initial attention matrix, and how does that matrix lead to rank collapse through depth or width? For a specified graph family and score model, we study how degree, sparsity, expansion and score variance determine the non-dominant singular spectrum of the graph-supported softmax matrix, then how this spectrum controls representation stable rank and loss of node variation. In a GAT, parameters and node features determine the scores, while the graph determines which scores compete, so neither component can be studied alone. We present results which isolate graph topology using uniform attention on two deterministic graph families. Write $s_1 \geq s_2$ for the two largest singular values of the post-softmax attention matrix. On the complete graph without self loops, $s_2/s_1 \to 0$ as $n$, the number of nodes, grows. The leading singular direction therefore dominates all the others and the attention operator approaches rank one. On the balanced double star, $s_2/s_1 = 1$ for every $n$, so the two leading directions remain equally strong and the spectral gap is absent. These cases show that graph topology alone can create or prevent an initial singular gap before learned score variation is introduced. Turning to the well-studied Erdos-Renyi random graphs as an initial random model, we first vary the score distribution. We then move to connected regular graphs and to the correlated scores of initialised GAT and GATv2 models [4], graph transformers and products through depth. At each stage, we relate the non-dominant spectrum to representation stable rank and node variation. This analysis turns the descriptive rank-collapse failure mode into a design principle for graph attention networks. By identifying which topological and score-statistical regimes create a harmful spectral gap, it could inform topology-aware initialisation or normalisation that controls score variance or suppresses a dominant direction without discarding useful connectivity. The broader goal is to let deeper and wider graph attention models combine long-range information while preserving the node distinctions needed for accurate predictions. References. [1] Roth and Liebig, "Rank Collapse Causes Over-Smoothing and Over-Correlation in Graph Neural Networks," LoG 2024. [2] Wu et al., "Demystifying Oversmoothing in Attention-Based Graph Neural Networks," NeurIPS 2023. [3] Nait Saada et al., "Mind the Gap: A Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers," ICML 2025. [4] Brody et al., "How Attentive Are Graph Attention Networks?" ICLR 2022.