Different Tokenization Schemes Lead to Comparable Performance in Spanish Number Agreement

Catherine Arnett, Tyler Chang, Sean Trott


Abstract
The relationship between language model tokenization and performance is an open area of research. Here, we investigate how different tokenization schemes impact number agreement in Spanish plurals. We find that morphologically-aligned tokenization performs similarly to other tokenization schemes, even when induced artificially for words that would not be tokenized that way during training. We then present exploratory analyses demonstrating that language model embeddings for different plural tokenizations have similar distributions along the embedding space axis that maximally distinguishes singular and plural nouns. Our results suggest that morphologically-aligned tokenization is a viable tokenization approach, and existing models already generalize some morphological patterns to new items. However, our results indicate that morphological tokenization is not strictly required for performance.
Anthology ID:
2024.sigmorphon-1.4
Volume:
Proceedings of the 21st SIGMORPHON workshop on Computational Research in Phonetics, Phonology, and Morphology
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Garrett Nicolai, Eleanor Chodroff, Frederic Mailhot, Çağrı Çöltekin
Venue:
SIGMORPHON
SIG:
SIGMORPHON
Publisher:
Association for Computational Linguistics
Note:
Pages:
32–38
Language:
URL:
https://aclanthology.org/2024.sigmorphon-1.4/
DOI:
10.18653/v1/2024.sigmorphon-1.4
Bibkey:
Cite (ACL):
Catherine Arnett, Tyler Chang, and Sean Trott. 2024. Different Tokenization Schemes Lead to Comparable Performance in Spanish Number Agreement. In Proceedings of the 21st SIGMORPHON workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 32–38, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
Different Tokenization Schemes Lead to Comparable Performance in Spanish Number Agreement (Arnett et al., SIGMORPHON 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.sigmorphon-1.4.pdf
Video:
 https://aclanthology.org/2024.sigmorphon-1.4.mp4