The Cleaning Layer: How the Convolution Bias on Task-Irrelevant Positions Inflates Curvature and Gradient Noise
Andreas Horlbeck ⋅ Adrian Krenzer
Abstract
The convolution bias places activations on background positions outside the task-relevant region. We propose the Cleaning Layer, a zero-parameter module removing background-induced activations at their source. For a 1D regression task, we derive the loss surface, gradient decomposition, Hessian eigenvalues, and convergence rate in closed form. Without Cleaning, the condition number and the bias-gradient variance grow as $1/(1-\rho)^2$ with the background fraction $\rho$. Two mechanisms separate: curvature governs the convergence rate and stability boundary, while gradient-noise amplification sets the stationary loss floor. With one shared learning rate, Cleaning attains a lower stationary loss under label noise and a wider range of stable learning rates. If $\rho$ is equal across samples, Cleaning under SGD is equivalent to scaling the bias learning rate and initialization by $(1-\rho)^2$ and $(1-\rho)$. If $\rho$ varies across samples, a model without Cleaning, with its bias learning rate rescaled by $\mathbb{E}[(1-\rho)^2]$, matches the stationary loss. When $\rho$ varies strongly, Cleaning still attains a lower minimum loss.
Chat is not available.
Successful Page Load