Poly-S: Analyzing and Improving Polytropon for Data-Efficient Multi-Task Learning

Polytropon learns a set of modular skills, which can be re-combined and fine-tuned on novel tasks with limited data. In this paper, we first investigate what makes this method successful. Specifically, we extend the evaluation benchmark to include more datasets and design a series of controlled experiments to isolate the impact of different components.We then propose a new method, Poly-S, which allows for a more fine-grained control over the combination of skills, with no additional cost in compute at inference time. We evaluate Poly-S on three multi-task NLP benchmarks, and observe improvements over strong baselines.