Evaluation of Deep Learning-Based Motor Imagery EEG Decoders Using Low-Cost EEG-Based Brain–Computer Interfaces

Motor imagery (MI)-based brain–computer interfaces (BCIs) have demonstrated potential for neurorehabilitation. Nonetheless, most decoder evaluations rely on research-grade EEG systems with high channel density and controlled acquisition, leaving it unclear whether current machine learning and deep learning-based MI decoders remain reliable under low-cost EEG hardware. Evaluation of decoder robustness under hardware constraints is essential for developing practical, accessible BCIs. This study presents a systematic benchmark of state-of-art deep learning-based MI EEG decoders using an open-access MI vs. Rest dataset recorded with an OpenBCI Cyton+Daisy board in a non-shielded office. The dataset comprises recordings from 10 healthy participants (four female, all right-handed, mean age 26.1 ± 4.0 years). Five representative classifiers are selected, specifically, CSP+LDA, EEGNet, ShallowConvNet, DeepConvNet, and EEGConformer. Models are evaluated across three validation scenarios: stratified within-subject evaluation, 5-fold within-session cross- validation, and leave-one-subject-out (LOSO) analysis. The effect of sliding-window augmentation is further examined. Metrics such as accuracy and Cohen’s kappa coefficient, as well as statistical comparisons using Friedman tests followed by Holm-corrected Wilcoxon signed-rank tests are used. Within-session evaluation shows CSP+LDA achieves the highest mean accuracy (77.9%, 95% bootstrap CI: 72.6–82.7), followed closely by EEGNet (75.9%, 95% bootstrap CI: 70.6–80.7), with no statistically significant difference after multiplicity correction. In contrast, LOSO evaluation identifies EEGNet as the best-performing decoder (64.3%, 95% bootstrap CI: 61.8–66.8), underscoring the difficulty of subject-independent decoding on low-cost hardware. Sliding-window augmentation improves EEGNet and EEGConformer by 13.4 and 16.8 percentage points, respectively, while offering limited benefit to CSP-based approaches. These findings show that decoder performance rankings shift across evaluation scenarios, and that strong within-subject performance does not guarantee cross-user generalization. This study establishes a reproducible benchmark for MI EEG decoding on consumer-grade hardware. Additionally, it demonstrates the need for prioritizing subject-independent robustness not just within- session accuracy, when designing MI decoders for deployable, low-cost BCI systems in real-world settings.