Killing Two Flies with One Stone: An Attempt to Break LLMs Using English-Icelandic Idioms and Proper Names

Bjarki Ármannsson, Hinrik Hafsteinsson, Atli Jasonarson, Steinthor Steingrimsson


Abstract
The submission of the Árni Magnússon Institute’s team to the WMT24 test suite subtask focuses on idiomatic expressions and proper names for the English→Icelandic translation direction. Intuitively and empirically, idioms and proper names are known to be a significant challenge for neural translation models. We create two different test suites. The first evaluates the competency of MT systems in translating common English idiomatic expressions, as well as testing whether systems can distinguish between those expressions and the same phrases when used in a literal context. The second test suite consists of place names that should be translated into their Icelandic exonyms (and correctly inflected) and pairs of Icelandic names that share a surface form between the male and female variants, so that incorrect translations impact meaning as well as readibility. The scores reported are relatively low, especially for idiomatic expressions and place names, and indicate considerable room for improvement.
Anthology ID:
2024.wmt-1.31
Volume:
Proceedings of the Ninth Conference on Machine Translation
Month:
November
Year:
2024
Address:
Miami, Florida, USA
Editors:
Barry Haddow, Tom Kocmi, Philipp Koehn, Christof Monz
Venue:
WMT
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
451–458
Language:
URL:
https://aclanthology.org/2024.wmt-1.31
DOI:
10.18653/v1/2024.wmt-1.31
Bibkey:
Cite (ACL):
Bjarki Ármannsson, Hinrik Hafsteinsson, Atli Jasonarson, and Steinthor Steingrimsson. 2024. Killing Two Flies with One Stone: An Attempt to Break LLMs Using English-Icelandic Idioms and Proper Names. In Proceedings of the Ninth Conference on Machine Translation, pages 451–458, Miami, Florida, USA. Association for Computational Linguistics.
Cite (Informal):
Killing Two Flies with One Stone: An Attempt to Break LLMs Using English-Icelandic Idioms and Proper Names (Ármannsson et al., WMT 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.wmt-1.31.pdf