Visualizing Loss Functions as Topological Landscape Profiles

In machine learning, a loss function measures the difference between model predictions and ground-truth (or target) values. Visualizing how this loss changes as a neural network's parameters are varied can provide insights into the local structure of the so-called loss landscape (e.g., smoothness) and global properties of the underlying model (e.g., generalization performance). While various methods for visualizing the loss landscape have been proposed, many approaches limit sampling to just one or two directions, ignoring potentially relevant information in this extremely high-dimensional space. This paper introduces a new representation based on topological data analysis that enables the visualization of higher dimensional loss landscapes. In addition to this new topological landscape profile representation, we present an interactive tool for users to explore these landscapes across different models and hyperparameters, enabling more systematic comparisons and informed model exploration. We highlight several use cases, including image segmentation (e.g., UNet) and scientific machine learning (e.g., physics-informed neural networks), showing how visualizing higher-dimensional loss landscapes can provide new insights into model performance and learning dynamics. Through these examples, we provide new insights into how loss landscapes vary across distinct hyperparameter spaces, finding that the topology of the loss landscape is simpler for better-performing models. Interestingly, we observe more variation in the shape of loss landscapes near transitions from low to high model performance.