Hand-Engineered Image-Computable Models Can Still Outperform DNNs in V1 Similarity

Task-based Deep Neural Network models (DNNs) are widely used as models of inferotemporal visual cortex (IT), with early work showing a large jump over previous hand-made models (Yamins & DiCarlo, 2014). However, recent work has suggested that over time, not only has the performance-alignment relationship plateaued but reversed, with high performing models becoming worse models of IT (Linsley et al., 2023). Here we attempt to see if this reversal extends to earlier cortical regions. We evaluate a broad set of models, including SOTA IT similarity models (Schrimpf et al., 2020), high-task-performant CNN's and VIT's, along with other models directly attempting to model V1 (Dapello et al., 2020). Along with DNNs we also test traditional, hand-crafted models such as HMAX (Riesenhuber & Poggio, 1999). Surprisingly, we find that the most modern models are equal or worse models of V1 compared to HMAX even as they are better models of IT. Furthermore, HMAX becomes the best model when utilizing representational similarity scores that care about representational geometry or strict pairwise matching, being only permutation invariant. These results suggest that further research with hand-crafted image computable models is required, as these models may still outperform our most modern models in certain circumstances.