Lightweight Text Classifier using Sinusoidal Positional Encoding

Byoung-Doo Oh, Yu-Seop Kim


Abstract
Large and complex models have recently been developed that require many parameters and much time to solve various problems in natural language processing. This paper explores an efficient way to avoid models being too complicated and ensure nearly equal performance to models showing the state-of-the-art. We propose a single convolutional neural network (CNN) using the sinusoidal positional encoding (SPE) in text classification. The SPE provides useful position information of a word and can construct a more efficient model architecture than before in a CNN-based approach. Our model can significantly reduce the parameter size (at least 67%) and training time (up to 85%) while maintaining similar performance to the CNN-based approach on multiple benchmark datasets.
Anthology ID:
2020.aacl-main.8
Volume:
Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing
Month:
December
Year:
2020
Address:
Suzhou, China
Editors:
Kam-Fai Wong, Kevin Knight, Hua Wu
Venue:
AACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
63–69
Language:
URL:
https://aclanthology.org/2020.aacl-main.8
DOI:
10.18653/v1/2020.aacl-main.8
Bibkey:
Cite (ACL):
Byoung-Doo Oh and Yu-Seop Kim. 2020. Lightweight Text Classifier using Sinusoidal Positional Encoding. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 63–69, Suzhou, China. Association for Computational Linguistics.
Cite (Informal):
Lightweight Text Classifier using Sinusoidal Positional Encoding (Oh & Kim, AACL 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.aacl-main.8.pdf
Data
MPQA Opinion Corpus