Contrast inversion reveals hierarchical asymmetries of contrast processing in biological and artificial vision

Contrast is a fundamental visual feature, encoded as early as the retina by segregated ON and OFF pathways. While these pathways are largely symmetric, subtle biases exist that shape perception and cortical responses. Here, we extend the study of contrast processing to color images across the hierarchy of the primate visual ventral stream and convolutional neural networks (CNNs). Using Neuropixels recordings from macaque V1 through IT, and contrast inversion we show that in a biological system contrast polarity is weakly encoded in early cortex but becomes stronger downstream, peaking in IT. Surprisingly, CNNs exhibit the opposite trend: contrast polarity is strongly represented in the first layer, lost in intermediate layers, and partially recovered later. Thus, early visual areas in the brain rely on local features symmetric to contrast inversion and this symmetry is broken in high-level visual areas. while the CNNs rely on asymmetric local and high-level features. This divergence reveals a fundamental asymmetry in how biological and artificial systems balance tolerance and sensitivity to strong out of distribution images, such as contrast inversion, as early as the first layer, providing new constraints for improving both neural models and machine vision.