Models that accurately predict properties based on chemical structure are valuable tools in drug discovery. However, for many properties, public and private training sets are typically small, and it is difficult for the models to generalize well outside of the training data. Recently, large language models have addressed this problem by using self-supervised pretraining on large unlabeled datasets, followed by fine-tuning on smaller, labeled datasets. In this paper, we report MolE, a molecular foundation model that adapts the DeBERTa architecture to be used on molecular graphs together with a two-step pretraining strategy. The first step of pretraining is a self-supervised approach focused on learning chemical structures, and the second step is a massive multi-task approach to learn biological information. We show that fine-tuning pretrained MolE achieves state-of-the-art results on 9 of the 22 ADMET tasks included in the Therapeutic Data Commons.
Oscar Méndez-Lucio (Recursion)
Christos Nicolaou (University of Cyprus)
Berton Earnshaw (Recursion Pharmaceuticals)
More from the Same Authors
2022 : Biological Cartography: Building and Benchmarking Representations of Life »
Safiye Celik · Jan-Christian Huetter · Sandra Melo · Nathan Lazar · Rahul Mohan · Conor Tillinghast · Tommaso Biancalani · Marta Fay · Berton Earnshaw · Imran Haque