A Call for Consistency in Reporting Typological Diversity

Wessel Poelman, Esther Ploeger, Miryam de Lhoneux, Johannes Bjerva


Abstract
In order to draw generalizable conclusions about the performance of multilingual models across languages, it is important to evaluate on a set of languages that captures linguistic diversity.Linguistic typology is increasingly used to justify language selection, inspired by language sampling in linguistics.However, justifications for ‘typological diversity’ exhibit great variation, as there seems to be no set definition, methodology or consistent link to linguistic typology.In this work, we provide a systematic insight into how previous work in the ACL Anthology uses the term ‘typological diversity’.Our two main findings are: 1) what is meant by typologically diverse language selection is not consistent and 2) the actual typological diversity of the language sets in these papers varies greatly.We argue that, when making claims about ‘typological diversity’, an operationalization of this should be included.A systematic approach that quantifies this claim, also with respect to the number of languages used, would be even better.
Anthology ID:
2024.sigtyp-1.10
Volume:
Proceedings of the 6th Workshop on Research in Computational Linguistic Typology and Multilingual NLP
Month:
March
Year:
2024
Address:
St. Julian's, Malta
Editors:
Michael Hahn, Alexey Sorokin, Ritesh Kumar, Andreas Shcherbakov, Yulia Otmakhova, Jinrui Yang, Oleg Serikov, Priya Rani, Edoardo M. Ponti, Saliha Muradoğlu, Rena Gao, Ryan Cotterell, Ekaterina Vylomova
Venues:
SIGTYP | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
75–77
Language:
URL:
https://aclanthology.org/2024.sigtyp-1.10
DOI:
Bibkey:
Cite (ACL):
Wessel Poelman, Esther Ploeger, Miryam de Lhoneux, and Johannes Bjerva. 2024. A Call for Consistency in Reporting Typological Diversity. In Proceedings of the 6th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, pages 75–77, St. Julian's, Malta. Association for Computational Linguistics.
Cite (Informal):
A Call for Consistency in Reporting Typological Diversity (Poelman et al., SIGTYP-WS 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.sigtyp-1.10.pdf