MetaMetrics-MT: Tuning Meta-Metrics for Machine Translation via Human Preference Calibration

David Anugraha, Garry Kuwanto, Lucky Susanto, Derry Tanti Wijaya, Genta Winata


Abstract
We present MetaMetrics-MT, an innovative metric designed to evaluate machine translation (MT) tasks by aligning closely with human preferences through Bayesian optimization with Gaussian Processes. MetaMetrics-MT enhances existing MT metrics by optimizing their correlation with human judgments. Our experiments on the WMT24 metric shared task dataset demonstrate that MetaMetrics-MT outperforms all existing baselines, setting a new benchmark for state-of-the-art performance in the reference-based setting. Furthermore, it achieves comparable results to leading metrics in the reference-free setting, offering greater efficiency.
Anthology ID:
2024.wmt-1.32
Volume:
Proceedings of the Ninth Conference on Machine Translation
Month:
November
Year:
2024
Address:
Miami, Florida, USA
Editors:
Barry Haddow, Tom Kocmi, Philipp Koehn, Christof Monz
Venue:
WMT
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
459–469
Language:
URL:
https://aclanthology.org/2024.wmt-1.32
DOI:
10.18653/v1/2024.wmt-1.32
Bibkey:
Cite (ACL):
David Anugraha, Garry Kuwanto, Lucky Susanto, Derry Tanti Wijaya, and Genta Winata. 2024. MetaMetrics-MT: Tuning Meta-Metrics for Machine Translation via Human Preference Calibration. In Proceedings of the Ninth Conference on Machine Translation, pages 459–469, Miami, Florida, USA. Association for Computational Linguistics.
Cite (Informal):
MetaMetrics-MT: Tuning Meta-Metrics for Machine Translation via Human Preference Calibration (Anugraha et al., WMT 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.wmt-1.32.pdf