Rigorous Evaluation of Large Language Models for Malaria Drug Discovery
Abstract
African research institutions bear 94% of the global malaria burden yet are systematically excluded from the AI tools transforming drug discovery. Closed-source models incur prohibitive API costs, while open-source LLMs require scarce GPU infrastructure. We introduce \textbf{Malaria-Instruct}, the first instruction-following dataset for antimalarial virtual screening (74K+ datapoints, curated from the ChEMBL Legacy Malaria corpus with rigorous deduplication, assay harmonisation, and negative augmentation), with rigorous out-of-distribution (Lo-Hi dissimilarity) splittingand systematically evaluate five open-source LLMs: Gemma-22B/9B, TxGemma2B/9B, and LlaSMol-Mistral7B, against classical ML baselines and frontier proprietary models (Gemini 2.5, OpenAI o3). \textbf{Domain-specific fine-tuning is non-negotiable:} Gemma-2-2B collapses from ROC-AUC0.706 under fine-tuning to 0.005 under few-shot prompting and neither Gemini 2.5 (best ROC-AUC 0.528) nor OpenAI o3 (best 0.591) achieves reliable discrimination without gradient-level adaptation. Fine-tuned open-source LLMs are within reach of resource-constrained groups: all training was conducted on Google Colab Pro+ (16--24GB GPUs, $50/month), with inference from 0.0043s/molecule (Gemma-2-2B) to 0.082s/molecule (LlaSMol-Mistral). We release all models, datasets, and Colab notebooks to lower the barrier for African researchers entering AI-driven drug discovery.