Beyond Decomposition: Characterizing Structure Across Learned Representation Components

Understanding how information is organized within learned representations remains challenging. Existing analyses primarily characterize representations through global geometric properties, while decomposition-based approaches often focus on the semantics or properties of individual components. We study a complementary level of analysis: the organization induced among components when a representation is expressed through a decomposition, which we refer to as decomposition structure. To study this object, we introduce NORD (Neural Organization in Representation Decompositions), a decomposition-independent framework that characterizes decomposition structure through complementary individual, relational, and global structural observations. We then ask whether decomposition structure differs systematically across learned representations under a controlled experimental design that separates variation due to representation source, decomposition configuration, and optimization. Using matched ViT-B/16 architectures, we compare representations learned with DINOv2 and MAE against an untrained ViT while holding architecture and decomposition procedure fixed. The resulting structural profiles reveal systematic differences across representation sources, showing that decomposition structure captures organizational properties of learned representations that complement conventional analyses of representation geometry.