Timezone: »

 
Poster
Memory Efficient Continual Learning with Transformers
Beyza Ermis · Giovanni Zappella · Martin Wistuba · Aditya Rawal · Cedric Archambeau

Wed Nov 30 02:00 PM -- 04:00 PM (PST) @ Hall J #608
in Poster Session 4 »

In many real-world scenarios, data to train machine learning models becomes available over time. Unfortunately, these models struggle to continually learn new concepts without forgetting what has been learnt in the past. This phenomenon is known as catastrophic forgetting and it is difficult to prevent due to practical constraints. For instance, the amount of data that can be stored or the computational resources that can be used might be limited. Moreover, applications increasingly rely on large pre-trained neural networks, such as pre-trained Transformers, since compute or data might not be available in sufficiently large quantities to practitioners to train from scratch. In this paper, we devise a method to incrementally train a model on a sequence of tasks using pre-trained Transformers and extending them with Adapters. Different than the existing approaches, our method is able to scale to a large number of tasks without significant overhead and allows sharing information across tasks. On both image and text classification tasks, we empirically demonstrate that our method maintains a good predictive performance without retraining the model or increasing the number of model parameters over time. The resulting model is also significantly faster at inference time compared to Adapter-based state-of-the-art methods.

Keywords: continual learning · transformers · image classification · BERT · ViT · text classification · adapters

Author Information

Beyza Ermis (Amazon)
Giovanni Zappella (Amazon Development Center Germany)
Martin Wistuba (Amazon)
Aditya Rawal (Amazon AWS AI LABS)
Cedric Archambeau (Amazon Web Services)

More from the Same Authors