GPT-2 Small Fine-Tuned on Logical Reasoning Summarizes Information on Punctuation Tokens
Sonakshi Chauhan · Atticus Geiger
Abstract
How is information stored and aggregated within a language model performing inference? Preliminary evidence suggests that representations of punctuation tokens might serve as ``summary points'' for information about preceding text. We add to this body of evidence by demonstrating that GPT-2 small fine-tuned on the RuleTaker logical inference dataset aggregates crucial information about rules and sentences above period tokens.
Chat is not available.
Successful Page Load