Speech Emotion Recognition with Classical and DeepLearning Models: Speaker-Independent Evaluation,Mechanistic Analysis, and an Accuracy–EfficiencyFrontier

Speech Emotion Recognition (SER) is commonly benchmarked with random stratified train/test splits, which we show allows speaker identity to leak between splits and substantially inflates reported accuracy. We replace this with 5-fold speaker-independent (GroupKFold) cross-validation on RAVDESS and CREMA- D, and run four further analyses on the corrected protocol. First, SVMs retain 92-99% of training samples as support vectors regardless of training-set size or regularization strength, on both datasets, indicating a structural limitation of MFCC features for margin-based classifiers. Second, probing an MLP's hidden- layer activations for speaker identity shows - counter to our initial hypothesis - that honestly-trained models encode speaker identity more strongly than leakage- trained models, suggesting leaky evaluation inflates accuracy through narrow shortcuts rather than richer representations. Third, cross-dataset generalization is weak (23-26% vs. 16.7% chance), indicating reliance on dataset-specific acoustic conditions and limited transferability across corpora. Fourth, we measure a real (not theoretical) accuracy-efficiency frontier from MFCC features through frozen and fine-tuned wav2vec2. Frozen wav2vec2 outperforms our MFCC+MLP pipeline by 11-13pp at 966-4,833x more parameters; fine-tuning closes a further 5-10pp. All findings are replicated across both datasets and, where feasible, across multiple folds, providing consistent evidence that evaluation protocol and representation choice substantially affect conclusions about SER performance.