Institute of Information Science, Academia Sinica

We introduce segmental recurrent neural networks (SRNNs) which define, given an input sequence, a joint probability distribution over segmentations of the input and labelings of the segments. Representations of the input segments (i.e., contiguous subsequences of the input) are computed by encoding their constituent tokens using bidirectional recurrent neural nets, and these “segment embeddings” are used to define compatibility scores with output labels. These local compatibility scores are integrated using a global semi-Markov conditional random field. Both fully supervised training—in which segment boundaries and labels are observed—as well as partially supervised training—in which segment boundaries are latent—are straightforward. Experiments on handwriting recognition and joint Chinese word segmentation/POS tagging show that, compared to models that do not explicitly represent segments such as BIO tagging schemes and connectionist temporal classification (CTC), SRNNs obtain substantially higher accuracies. We also performed experiments on the TIMIT dataset. where we achieved 17.3% phone error rate (PER) from the first-pass decoding — the best reported result using CRFs, despite the fact that we only used a zeroth-order CRF and without using any language model.

Lingpeng Kong is a Ph.D. candidate in School of Computer Science, Carnegie Mellon University, co-advised by Prof. Noah Smith and Prof. Chris Dyer. His main research interests are in designing algorithms to tackle the core problems in natural language processing (NLP). His work utilizes methods from machine learning, optimization and combinatorial algorithms with applications related to syntactic parsing, machine translation, and social media. Prior to CMU, he worked in IBM China Systems and Technology Lab.