Different Paths to Authenticity: Vision Transformers and ResNet Learn Distinct Cues for Real vs. Generated Image Classification

Advancements in AI image generation have led to cheaper, faster, and highly personalized prototyping of creative assets across industries. However, AI has also raised concerns about misinformation and the erosion of trust in visual evidence. Thus, in this work, we study two deep learning architectures - ResNet-50 and ViT-B/16 - for distinguishing real from AI-generated images (AIGIs). We train and evaluate both architectures on the MS COCOAI dataset, which includes AIGIs from five distinct generators: Stable Diffusion 2.1, Stable Diffusion XL, Stable Diffusion 3, DALL-E 3, and MidJourney v6. When employing transfer learning from ImageNet pre-trained weights, ViT-B/16 achieves 96.65\% accuracy and 94.34\% macro F1 score, while ResNet-50 reaches 95.34\% accuracy and 92.39\% macro F1 score. Nevertheless, when fine-tuned on images from individual generators rather than the whole dataset, both architectures show limited cross-generator transferability, struggling to detect AIGIs from unseen generator models. To further investigate the features that each architecture learns and relies on for classification, we also utilized style transfer by applying real-image style to generated images. The results show that ResNet-50 relies primarily on high-frequency texture cues, while ViT relies on low-frequency color signals for detection.