Measuring few-shot extrapolation with program induction

Neural networks are capable of learning complex functions, but still have problems generalizing from few examples and beyond their training distribution. Meta-learning provides a paradigm to train networks to learn from few examples, but it has been shown that some of its most popular benchmarks do not require significant adaptation to each task nor learning representations that extrapolate beyond the training distribution. Program induction lies at the opposite end of the spectrum: programs are capable of extrapolating from very few examples, but we still do not know how to efficiently search these discrete spaces. We propose a common benchmark for both communities, by learning to extrapolate from few examples coming from the execution of small programs. These are obtained by leveraging a C++ interpreter on codes from programming competitions and extracting small sub-codes with their corresponding input-output pairs. Statistical analysis and preliminary human experiments show the potential of this benchmark for enabling progress in few-shot extrapolation.