ASD-FER: An Annotated Multi-Class Benchmark and Comparative Study for Facial Emotion Recognition in Children With Autism Spectrum Disorder

Children with Autism Spectrum Disorder (ASD) often produce atypical, ambiguous, or reduced facial expressions, limiting both clinical observation and automated Facial Emotion Recognition (FER) systems trained on neurotypical faces. Progress on ASD-specific FER is hindered by atypical Action-Unit configurations, severe class imbalance (positive emotions vastly outnumber fear, sadness, and surprise), and very small labeled datasets. No prior study has systematically compared transformer architectures against CNN baselines under one controlled protocol, paired with domain-specific pre-training analysis. We introduce ASD-FER, a benchmark of 2,779 ASD facial images spanning six emotion categories (angry, fear, happy, neutral, sad, surprise), each labeled by three annotators via majority voting. Images were face-detected, cropped, and resized to $224\times224$ pixels, then augmented using eight techniques, yielding 25,011 images. StratifiedGroupKFold was applied at the original-image level to prevent augmentation leakage across folds. We benchmark nine models under stratified 10-fold cross-validation and an independent 90/10 held-out evaluation: a linear SVM, a from-scratch CNN, four ImageNet-pretrained CNNs (AlexNet, ResNet34, ResNet50, MobileNetV2, further pretrained on combined ASD-emotion corpora), and three transformer backbones sharing one downstream architecture --- an Emotion Region Enhancement gate, six learnable emotion query tokens, a two-layer transformer decoder, and a supervised contrastive objective: a compact Hybrid CNN-Transformer (2.66M parameters, domain-specific pre-training), Swin-T (44.7M, ImageNet-1K), and ViT (102.95M, ImageNet-21K). On the held-out 90/10 split, ViT achieved the highest accuracy (91.01\%), while Swin-T obtained the best balanced accuracy (0.7447) and macro-F1 (0.7078); the lightweight Hybrid CNN-Transformer stayed competitive (89.57\% accuracy, 0.6924 macro-F1). Among conventional baselines, ResNet50 led on both accuracy (87.05\%) and macro-F1, while AlexNet collapsed to majority-class prediction. Under 10-fold cross-validation, transformer models exceeded the best CNN baseline by 1.7--3.4 accuracy points and 0.07--0.08 macro-F1. All models struggled on minority classes (fear, sad, surprise; jointly 9\% of samples), with F1-scores of 0.40--0.67 even for the best transformer models. ASD-FER is, to our knowledge, the first multi-class (six-emotion) ASD facial-expression benchmark at this scale, providing the first controlled comparison of hybrid, hierarchical-window, and global-attention transformer paradigms against classical/CNN baselines for ASD FER~\cite{kopalidis2024advances, narayan2025facexformer}, with an efficiency--accuracy trade-off analysis. Limitations include a single, non-clinically-verified source dataset and persistent minority-class under-recognition; future work targets clinically recruited data, cross-dataset generalization, and multimodal extensions.