Skip to yearly menu bar Skip to main content

Workshop: 4th Workshop on Self-Supervised Learning: Theory and Practice

BarcodeBERT: Transformers for Biodiversity Analysis

Pablo Millan Arias · Niousha Sadjadi · Monireh Safari · ZeMing Gong · Austin T. Wang · Scott Lowe · Joakim Bruslund Haurum · Iuliia Zarubiieva · Dirk Steinke · Lila Kari · Angel Chang · Graham Taylor


Understanding biodiversity is a global challenge, in which DNA barcodes—short snippets of DNA that cluster by species—play a pivotal role. In particular, invertebrates, a highly diverse and under-explored group, pose unique taxonomic complexities. We explore machine learning approaches, comparing supervised CNNs, fine-tuned foundation models, and a DNA barcode-specific masking strategy across datasets of varying complexity. While simpler datasets and tasks favor supervised CNNs or fine-tuned transformers, challenging species-level identification demands a paradigm shift towards self-supervised pretraining. We propose BarcodeBERT, the first self-supervised method for general biodiversity analysis, leveraging a 1.5M invertebrate DNA barcode reference library. This work highlights how dataset specifics and coverage impact model selection, and underscores the role of self-supervised pretraining in achieving high-accuracy DNA barcode-based identification at the species and genus level. Indeed, without the fine-tuning step, BarcodeBERT pretrained on a large DNA barcode dataset outperforms DNABERT and DNABERT-2 on multiple downstream classification tasks. The code repository is available at

Chat is not available.