Shun Kiyono
2020
Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction
Masahiro Kaneko
|
Masato Mita
|
Shun Kiyono
|
Jun Suzuki
|
Kentaro Inui
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
This paper investigates how to effectively incorporate a pre-trained masked language model (MLM), such as BERT, into an encoder-decoder (EncDec) model for grammatical error correction (GEC). The answer to this question is not as straightforward as one might expect because the previous common methods for incorporating a MLM into an EncDec model have potential drawbacks when applied to GEC. For example, the distribution of the inputs to a GEC model can be considerably different (erroneous, clumsy, etc.) from that of the corpora used for pre-training MLMs; however, this issue is not addressed in the previous methods. Our experiments show that our proposed method, where we first fine-tune a MLM with a given GEC corpus and then use the output of the fine-tuned MLM as additional features in the GEC model, maximizes the benefit of the MLM. The best-performing model achieves state-of-the-art performances on the BEA-2019 and CoNLL-2014 benchmarks. Our code is publicly available at: https://github.com/kanekomasahiro/bert-gec.
ESPnet-ST: All-in-One Speech Translation Toolkit
Hirofumi Inaguma
|
Shun Kiyono
|
Kevin Duh
|
Shigeki Karita
|
Nelson Yalta
|
Tomoki Hayashi
|
Shinji Watanabe
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations
We present ESPnet-ST, which is designed for the quick development of speech-to-speech translation systems in a single framework. ESPnet-ST is a new project inside end-to-end speech processing toolkit, ESPnet, which integrates or newly implements automatic speech recognition, machine translation, and text-to-speech functions for speech translation. We provide all-in-one recipes including data pre-processing, feature extraction, training, and decoding pipelines for a wide range of benchmark datasets. Our reproducible results can match or even outperform the current state-of-the-art performances; these pre-trained models are downloadable. The toolkit is publicly available at https://github.com/espnet/espnet.
Search
Co-authors
- Masahiro Kaneko 1
- Masato Mita 1
- Jun Suzuki 1
- Kentaro Inui 1
- Hirofumi Inaguma 1
- show all...
Venues
- ACL2