Timezone: »

 
Learning to Reason and Memorize with Self-Questioning
Jack Lanchantin · Shubham Toshniwal · Jason E Weston · arthur szlam · Sainbayar Sukhbaatar

@
in Memory in Artificial and Real Intelligence (MemARI) »
Event URL: https://openreview.net/forum?id=koaxgmGG0mX »

Large language models have been shown to struggle with limited context memory and multi-step reasoning [1]. We propose a simple method for solving both of these problems by allowing the model to ask questions and answer them. Unlike recent scratchpad approaches, the model can deviate from the input context at any time for self-questioning. This allows the model to recall information and perform reasoning on the fly as it reads the context, thus extending its memory and enabling multi-step reasoning. Our experiments on two synthetic tasks demonstrate that our method can successfully generalize to more complicated instances from their training setup by performing self-questioning at inference time.

Author Information

Jack Lanchantin (FAIR)
Shubham Toshniwal (FAIR, Meta AI)
Jason E Weston (Meta AI)

Jason Weston received a PhD. (2000) from Royal Holloway, University of London under the supervision of Vladimir Vapnik. From 2000 to 2002, he was a researcher at Biowulf technologies, New York, applying machine learning to bioinformatics. From 2002 to 2003 he was a research scientist at the Max Planck Institute for Biological Cybernetics, Tuebingen, Germany. From 2004 to June 2009 he was a research staff member at NEC Labs America, Princeton. From July 2009 onwards he has been a research scientist at Google, New York. Jason Weston's current research focuses on various aspects of statistical machine learning and its applications, particularly in text and images.

arthur szlam (Facebook)
Sainbayar Sukhbaatar (Meta AI)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors