Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance?

Ahmed Alajrami, Katerina Margatina, Nikolaos Aletras


Abstract
Understanding how and what pre-trained language models (PLMs) learn about language is an open challenge in natural language processing. Previous work has focused on identifying whether they capture semantic and syntactic information, and how the data or the pre-training objective affects their performance. However, to the best of our knowledge, no previous work has specifically examined how information loss in input token characters affects the performance of PLMs. In this study, we address this gap by pre-training language models using small subsets of characters from individual tokens. Surprisingly, we find that pre-training even under extreme settings, i.e. using only one character of each token, the performance retention in standard NLU benchmarks and probing tasks compared to full-token models is high. For instance, a model pre-trained only on single first characters from tokens achieves performance retention of approximately 90% and 77% of the full-token model in SuperGLUE and GLUE tasks, respectively.
Anthology ID:
2023.emnlp-main.563
Volume:
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
Month:
December
Year:
2023
Address:
Singapore
Editors:
Houda Bouamor, Juan Pino, Kalika Bali
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
9085–9108
Language:
URL:
https://aclanthology.org/2023.emnlp-main.563
DOI:
10.18653/v1/2023.emnlp-main.563
Bibkey:
Cite (ACL):
Ahmed Alajrami, Katerina Margatina, and Nikolaos Aletras. 2023. Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance?. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9085–9108, Singapore. Association for Computational Linguistics.
Cite (Informal):
Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance? (Alajrami et al., EMNLP 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.emnlp-main.563.pdf
Video:
 https://aclanthology.org/2023.emnlp-main.563.mp4