Timezone: »

 
LMPriors: Pre-Trained Language Models as Task-Specific Priors
Kristy Choi · Chris Cundy · Sanjari Srivastava · Stefano Ermon

@
in Foundation Models for Decision Making »
Event URL: https://openreview.net/forum?id=U2MnmJ7Sa4 »

Particularly in low-data regimes, an outstanding challenge in machine learning is developing principled techniques for augmenting our models with suitable priors. This is to encourage them to learn in ways that are compatible with our understanding of the world. But in contrast to generic priors such as shrinkage or sparsity, we draw inspiration from the recent successes of large-scale language models (LMs) to construct \emph{task-specific priors} distilled from the rich knowledge of LMs. Our method, Language Model Priors (LMPriors), incorporates auxiliary natural language metadata about the task---such as variable names and descriptions---to encourage downstream model outputs to be consistent with the LM's common-sense reasoning based on the metadata. Empirically, we demonstrate that LMPriors improve model performance in settings where such natural language descriptions are available, and perform well on several tasks that benefit from such prior knowledge, such as feature selection, causal inference, and safe reinforcement learning.

Author Information

Kristy Choi (Stanford University)
Chris Cundy (Stanford University)
Sanjari Srivastava (Stanford University)
Stefano Ermon (Stanford)

More from the Same Authors