Weight Space Learning needs to unify benchmarking! A taxonomy of evaluation practices

Weight-space learning (WSL) is a new branch of machine learning research that defines the observation space of a learning model over a dataset of neural network parameters. This position paper argues that the recently established community around WSL research needs to unify benchmarking practices, as the current benchmarking landscape is fragmented, obscuring progress assessment within the field. Frequent claims of novelty (often tied to new downstream tasks) and evaluations on divergent model populations (datasets in WSL) make systematic comparison difficult. We reveal this fragmentation through a structured literature analysis and argue for more consistent and comparable evaluation standards across methods.