Timezone: »
On-line handwriting recognition is unusual among sequence labelling tasks in that the underlying generator of the observed data, i.e. the movement of the pen, is recorded directly. However, the raw data can be difficult to interpret because each letter is spread over many pen locations. As a consequence, sophisticated pre-processing is required to obtain inputs suitable for conventional sequence labelling algorithms, such as HMMs. In this paper we describe a system capable of directly transcribing raw on-line handwriting data. The system consists of a recurrent neural network trained for sequence labelling, combined with a probabilistic language model. In experiments on an unconstrained on-line database, we record excellent results using either raw or pre-processed data, well outperforming a benchmark HMM in both cases.
Author Information
Alex Graves (Google DeepMind)
Main contributions to neural networks include the Connectionist Temporal Classification training algorithm (widely used for speech, handwriting and gesture recognition, e.g. by Google voice search), a type of differentiable attention for RNNs (originally for handwriting generation, now a standard tool in computer vision, machine translation and elsewhere), stochastic gradient variational inference, and Neural Turing Machines. He works at Google Deep Mind.
Santiago Fernandez (IDSIA)
Marcus Liwicki (Institute of Computer Science & Applied Math, University of Bern, Switzerland)
Horst Bunke (Institute of Computer Science & Applied Math, University of Bern, Switzerland)
Jürgen Schmidhuber (Swiss AI Lab, IDSIA (USI & SUPSI) - NNAISENSE)
Since age 15, his main goal has been to build an Artificial Intelligence smarter than himself, then retire. The Deep Learning Artificial Neural Networks developed since 1991 by his research groups have revolutionised handwriting recognition, speech recognition, machine translation, image captioning, and are now available to billions of users through Google, Microsoft, IBM, Baidu, and many other companies (DeepMind also was heavily influenced by his lab). His team's Deep Learners were the first to win object detection and image segmentation contests, and achieved the world's first superhuman visual classification results, winning nine international competitions in machine learning & pattern recognition. His formal theory of fun & creativity & curiosity explains art, science, music, and humor. He has published 333 papers, earned 7 best paper/best video awards, the 2013 Helmholtz Award of the International Neural Networks Society, and the 2016 IEEE Neural Networks Pioneer Award. He is also president of NNAISENSE, which aims at building the first practical general purpose AI.
More from the Same Authors
-
2019 Poster: Are Disentangled Representations Helpful for Abstract Visual Reasoning? »
Sjoerd van Steenkiste · Francesco Locatello · Jürgen Schmidhuber · Olivier Bachem -
2018 Poster: Recurrent World Models Facilitate Policy Evolution »
David Ha · Jürgen Schmidhuber -
2018 Oral: Recurrent World Models Facilitate Policy Evolution »
David Ha · Jürgen Schmidhuber -
2018 Poster: Learning to Reason with Third Order Tensor Products »
Imanol Schlag · Jürgen Schmidhuber -
2017 Poster: Neural Expectation Maximization »
Klaus Greff · Sjoerd van Steenkiste · Jürgen Schmidhuber -
2016 Symposium: Recurrent Neural Networks and Other Machines that Learn Algorithms »
Jürgen Schmidhuber · Sepp Hochreiter · Alex Graves · Rupesh K Srivastava -
2016 Poster: Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes »
Jack Rae · Jonathan J Hunt · Ivo Danihelka · Tim Harley · Andrew Senior · Gregory Wayne · Alex Graves · Timothy Lillicrap -
2016 Poster: Conditional Image Generation with PixelCNN Decoders »
Aaron van den Oord · Nal Kalchbrenner · Lasse Espeholt · koray kavukcuoglu · Oriol Vinyals · Alex Graves -
2016 Poster: Tagger: Deep Unsupervised Perceptual Grouping »
Klaus Greff · Antti Rasmus · Mathias Berglund · Hotloo Xiranood · Harri Valpola · Jürgen Schmidhuber -
2016 Poster: Memory-Efficient Backpropagation Through Time »
Audrunas Gruslys · Remi Munos · Ivo Danihelka · Marc Lanctot · Alex Graves -
2016 Poster: Strategic Attentive Writer for Learning Macro-Actions »
Alexander (Sasha) Vezhnevets · Volodymyr Mnih · Simon Osindero · Alex Graves · Oriol Vinyals · John Agapiou · koray kavukcuoglu -
2015 Poster: Training Very Deep Networks »
Rupesh K Srivastava · Klaus Greff · Jürgen Schmidhuber -
2015 Spotlight: Training Very Deep Networks »
Rupesh K Srivastava · Klaus Greff · Jürgen Schmidhuber -
2015 Poster: Parallel Multi-Dimensional LSTM, With Application to Fast Biomedical Volumetric Image Segmentation »
Marijn F Stollenga · Wonmin Byeon · Marcus Liwicki · Jürgen Schmidhuber -
2014 Poster: Recurrent Models of Visual Attention »
Volodymyr Mnih · Nicolas Heess · Alex Graves · koray kavukcuoglu -
2014 Spotlight: Recurrent Models of Visual Attention »
Volodymyr Mnih · Nicolas Heess · Alex Graves · koray kavukcuoglu -
2014 Poster: Deep Networks with Internal Selective Attention through Feedback Connections »
Marijn F Stollenga · Jonathan Masci · Faustino Gomez · Jürgen Schmidhuber -
2013 Poster: Compete to Compute »
Rupesh K Srivastava · Jonathan Masci · Sohrob Kazerounian · Faustino Gomez · Jürgen Schmidhuber -
2012 Poster: Deep Neural Networks Segment Neuronal Membranes in Electron Microscopy Images »
Dan Ciresan · Alessandro Giusti · luca Maria Gambardella · Jürgen Schmidhuber -
2011 Poster: Practical Variational Inference for Neural Networks »
Alex Graves -
2011 Spotlight: Practical Variational Inference for Neural Networks »
Alex Graves -
2010 Poster: Improving the Asymptotic Performance of Markov Chain Monte-Carlo by Inserting Vortices »
Yi Sun · Faustino Gomez · Jürgen Schmidhuber -
2008 Poster: Offline Handwriting Recognition with Multidimensional Recurrent Neural Networks »
Alex Graves · Jürgen Schmidhuber -
2008 Spotlight: Offline Handwriting Recognition with Multidimensional Recurrent Neural Networks »
Alex Graves · Jürgen Schmidhuber