BarcodeMamba: State Space Models for Biodiversity Analysis

DNA barcodes are crucial in biodiversity analysis for building automatic identification systems that recognize known species and discover unseen species. Unlike human genome modeling, barcode-based invertebrate identification poses challenges in the vast diversity of species and taxonomic complexities. Among Transformer-based foundation models, BarcodeBERT excelled in species-level identification of invertebrates, highlighting the effectiveness of self-supervised pretraining on barcode-specific datasets. Recently, structured state space models (SSMs) have emerged, with a time complexity that scales sub-quadratically with the context length. SSMs provide an efficient parameterization of sequence modeling relative to attention-based architectures. Given the success of Mamba and Mamba-2 in natural language, we designed BarcodeMamba, a performant and efficient foundation model for DNA barcodes in biodiversity analysis. We conducted a comprehensive ablation study on the impacts of self-supervised training and tokenization methods, and compared both versions of Mamba layers in terms of expressiveness and their capacity to identify "unseen" species held back from training. Our study shows that BarcodeMamba has competitive performance with BarcodeBERT even when using only 2.5\% as many parameters, and improves accuracy to 98.42\% on species-level linear probing without fine-tuning. In our scaling study, BarcodeMamba with 34\% of BarcodeBERT's parameters achieved 99.35\% species-level accuracy in linear probing for "seen" species and 60.38\% genus-level accuracy in 1-NN probing for unseen species. The code repository to reproduce our experiments is available at [URL available after blind review].