Efficient Automatic Punctuation Restoration Using Bidirectional Transformers with Robust Inference

Maury Courtland, Adam Faulkner, Gayle McElvain


Abstract
Though people rarely speak in complete sentences, punctuation confers many benefits to the readers of transcribed speech. Unfortunately, most ASR systems do not produce punctuated output. To address this, we propose a solution for automatic punctuation that is both cost efficient and easy to train. Our solution benefits from the recent trend in fine-tuning transformer-based language models. We also modify the typical framing of this task by predicting punctuation for sequences rather than individual tokens, which makes for more efficient training and inference. Finally, we find that aggregating predictions across multiple context windows improves accuracy even further. Our best model achieves a new state of the art on benchmark data (TED Talks) with a combined F1 of 83.9, representing a 48.7% relative improvement (15.3 absolute) over the previous state of the art.
Anthology ID:
2020.iwslt-1.33
Volume:
Proceedings of the 17th International Conference on Spoken Language Translation
Month:
July
Year:
2020
Address:
Online
Editors:
Marcello Federico, Alex Waibel, Kevin Knight, Satoshi Nakamura, Hermann Ney, Jan Niehues, Sebastian Stüker, Dekai Wu, Joseph Mariani, Francois Yvon
Venue:
IWSLT
SIG:
SIGSLT
Publisher:
Association for Computational Linguistics
Note:
Pages:
272–279
Language:
URL:
https://aclanthology.org/2020.iwslt-1.33
DOI:
10.18653/v1/2020.iwslt-1.33
Bibkey:
Cite (ACL):
Maury Courtland, Adam Faulkner, and Gayle McElvain. 2020. Efficient Automatic Punctuation Restoration Using Bidirectional Transformers with Robust Inference. In Proceedings of the 17th International Conference on Spoken Language Translation, pages 272–279, Online. Association for Computational Linguistics.
Cite (Informal):
Efficient Automatic Punctuation Restoration Using Bidirectional Transformers with Robust Inference (Courtland et al., IWSLT 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.iwslt-1.33.pdf
Video:
 http://slideslive.com/38929594