Fast and Accurate Deep Bidirectional Language Representations for Unsupervised Learning

Joongbo Shin, Yoonhyung Lee, Seunghyun Yoon, Kyomin Jung


Abstract
Even though BERT has achieved successful performance improvements in various supervised learning tasks, BERT is still limited by repetitive inferences on unsupervised tasks for the computation of contextual language representations. To resolve this limitation, we propose a novel deep bidirectional language model called a Transformer-based Text Autoencoder (T-TA). The T-TA computes contextual language representations without repetition and displays the benefits of a deep bidirectional architecture, such as that of BERT. In computation time experiments in a CPU environment, the proposed T-TA performs over six times faster than the BERT-like model on a reranking task and twelve times faster on a semantic similarity task. Furthermore, the T-TA shows competitive or even better accuracies than those of BERT on the above tasks. Code is available at https://github.com/joongbo/tta.
Anthology ID:
2020.acl-main.76
Volume:
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2020
Address:
Online
Editors:
Dan Jurafsky, Joyce Chai, Natalie Schluter, Joel Tetreault
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
823–835
Language:
URL:
https://aclanthology.org/2020.acl-main.76
DOI:
10.18653/v1/2020.acl-main.76
Bibkey:
Cite (ACL):
Joongbo Shin, Yoonhyung Lee, Seunghyun Yoon, and Kyomin Jung. 2020. Fast and Accurate Deep Bidirectional Language Representations for Unsupervised Learning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 823–835, Online. Association for Computational Linguistics.
Cite (Informal):
Fast and Accurate Deep Bidirectional Language Representations for Unsupervised Learning (Shin et al., ACL 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.acl-main.76.pdf
Video:
 http://slideslive.com/38928923
Code
 joongbo/tta
Data
GLUELibriSpeech