Timezone: »
Research on continual learning has led to a variety of approaches to mitigating catastrophic forgetting in feed-forward classification networks. Until now surprisingly little attention has been focused on continual learning of recurrent models applied to problems like image captioning. In this paper we take a systematic look at continual learning of LSTM-based models for image captioning. We propose an attention-based approach that explicitly accommodates the transient nature of vocabularies in continual image captioning tasks -- i.e. that task vocabularies are not disjoint. We call our method Recurrent Attention to Transient Tasks (RATT), and also show how to adapt continual learning approaches based on weight regularization and knowledge distillation to recurrent continual learning problems. We apply our approaches to incremental image captioning problem on two new continual learning benchmarks we define using the MS-COCO and Flickr30 datasets. Our results demonstrate that RATT is able to sequentially learn five captioning tasks while incurring no forgetting of previously learned ones.
Author Information
Riccardo Del Chiaro (University of Florence)
Bartłomiej Twardowski (Computer Vision Center, UAB)
Andrew Bagdanov (University of Florence)
Joost van de Weijer (Computer Vision Center Barcelona)
More from the Same Authors
-
2022 : Towards Informed Design and Validation Assistance in Computer Games Using Imitation Learning »
Alessandro Sestini · Carl Joakim Bergdahl · Konrad Tollmar · Andrew Bagdanov · Linus Gisslén -
2023 Poster: FeCAM: Exploiting the Heterogeneity of Class Distributions in Exemplar-Free Continual Learning »
Dipam Goswami · Yuyang Liu · Bartłomiej Twardowski · Joost van de Weijer -
2023 Poster: Dynamic Prompt Learning: Addressing Cross-Attention Leakage for Text-Based Image Editing »
kai wang · Fei Yang · Shiqi Yang · Muhammad Atif Butt · Joost van de Weijer -
2022 Workshop: Vision Transformers: Theory and applications »
Fahad Shahbaz Khan · Gul Varol · Salman Khan · Ping Luo · Rao Anwer · Ashish Vaswani · Hisham Cholakkal · Niki Parmar · Joost van de Weijer · Mubarak Shah -
2022 Spotlight: Lightning Talks 1B-4 »
Andrei Atanov · Shiqi Yang · Wanshan Li · Yongchang Hao · Ziquan Liu · Jiaxin Shi · Anton Plaksin · Jiaxiang Chen · Ziqi Pan · yaxing wang · Yuxin Liu · Stepan Martyanov · Alessandro Rinaldo · Yuhao Zhou · Li Niu · Qingyuan Yang · Andrei Filatov · Yi Xu · Liqing Zhang · Lili Mou · Ruomin Huang · Teresa Yeo · kai wang · Daren Wang · Jessica Hwang · Yuanhong Xu · Qi Qian · Hu Ding · Michalis Titsias · Shangling Jui · Ajay Sohmshetty · Lester Mackey · Joost van de Weijer · Hao Li · Amir Zamir · Xiangyang Ji · Antoni Chan · Rong Jin -
2022 Spotlight: Attracting and Dispersing: A Simple Approach for Source-free Domain Adaptation »
Shiqi Yang · yaxing wang · kai wang · Shangling Jui · Joost van de Weijer -
2022 Poster: Attracting and Dispersing: A Simple Approach for Source-free Domain Adaptation »
Shiqi Yang · yaxing wang · kai wang · Shangling Jui · Joost van de Weijer -
2021 Poster: Exploiting the Intrinsic Neighborhood Structure for Source-free Domain Adaptation »
Shiqi Yang · yaxing wang · Joost van de Weijer · Luis Herranz · Shangling Jui -
2020 Poster: DeepI2I: Enabling Deep Hierarchical Image-to-Image Translation by Transferring from GANs »
yaxing wang · Lu Yu · Joost van de Weijer -
2018 Poster: Image-to-image translation for cross-domain disentanglement »
Abel Gonzalez-Garcia · Joost van de Weijer · Yoshua Bengio -
2018 Poster: Memory Replay GANs: Learning to Generate New Categories without Forgetting »
Chenshen Wu · Luis Herranz · Xialei Liu · yaxing wang · Joost van de Weijer · Bogdan Raducanu -
2011 Poster: Portmanteau Vocabularies for Multi-Cue Image Representation »
Fahad S Khan · Joost van de Weijer · Andrew D Bagdanov · Maria Vanrell