Skip to yearly menu bar Skip to main content


Poster

Deep State Space Models for Unconditional Word Generation

Florian Schmidt · Thomas Hofmann

Room 210 #61

Keywords: [ Natural Language Processing ] [ Latent Variable Models ]


Abstract:

Autoregressive feedback is considered a necessity for successful unconditional text generation using stochastic sequence models. However, such feedback is known to introduce systematic biases into the training process and it obscures a principle of generation: committing to global information and forgetting local nuances. We show that a non-autoregressive deep state space model with a clear separation of global and local uncertainty can be built from only two ingredients: An independent noise source and a deterministic transition function. Recent advances on flow-based variational inference can be used to train an evidence lower-bound without resorting to annealing, auxiliary losses or similar measures. The result is a highly interpretable generative model on par with comparable auto-regressive models on the task of word generation.

Live content is unavailable. Log in and register to view live content