<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="von-daniken-etal-2022-effectiveness">
    <titleInfo>
        <title>On the Effectiveness of Automated Metrics for Text Generation Systems</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Pius</namePart>
        <namePart type="family">von Däniken</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jan</namePart>
        <namePart type="family">Deriu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Don</namePart>
        <namePart type="family">Tuggener</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Mark</namePart>
        <namePart type="family">Cieliebak</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2022</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yoav</namePart>
            <namePart type="family">Goldberg</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Zornitsa</namePart>
            <namePart type="family">Kozareva</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yue</namePart>
            <namePart type="family">Zhang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Abu Dhabi, United Arab Emirates</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>A major challenge in the field of Text Generation is evaluation, because we lack a sound theory that can be leveraged to extract guidelines for evaluation campaigns. In this work, we propose a first step towards such a theory that incorporates different sources of uncertainty, such as imperfect automated metrics and insufficiently sized test sets. The theory has practical applications, such as determining the number of samples needed to reliably distinguish the performance of a set of Text Generation systems in a given setting. We showcase the application of the theory on the WMT 21 and Spot-The-Bot evaluation data and outline how it can be leveraged to improve the evaluation protocol regarding the reliability, robustness, and significance of the evaluation outcome.</abstract>
    <identifier type="citekey">von-daniken-etal-2022-effectiveness</identifier>
    <identifier type="doi">10.18653/v1/2022.findings-emnlp.108</identifier>
    <location>
        <url>https://aclanthology.org/2022.findings-emnlp.108/</url>
    </location>
    <part>
        <date>2022-12</date>
        <extent unit="page">
            <start>1503</start>
            <end>1522</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T On the Effectiveness of Automated Metrics for Text Generation Systems
%A von Däniken, Pius
%A Deriu, Jan
%A Tuggener, Don
%A Cieliebak, Mark
%Y Goldberg, Yoav
%Y Kozareva, Zornitsa
%Y Zhang, Yue
%S Findings of the Association for Computational Linguistics: EMNLP 2022
%D 2022
%8 December
%I Association for Computational Linguistics
%C Abu Dhabi, United Arab Emirates
%F von-daniken-etal-2022-effectiveness
%X A major challenge in the field of Text Generation is evaluation, because we lack a sound theory that can be leveraged to extract guidelines for evaluation campaigns. In this work, we propose a first step towards such a theory that incorporates different sources of uncertainty, such as imperfect automated metrics and insufficiently sized test sets. The theory has practical applications, such as determining the number of samples needed to reliably distinguish the performance of a set of Text Generation systems in a given setting. We showcase the application of the theory on the WMT 21 and Spot-The-Bot evaluation data and outline how it can be leveraged to improve the evaluation protocol regarding the reliability, robustness, and significance of the evaluation outcome.
%R 10.18653/v1/2022.findings-emnlp.108
%U https://aclanthology.org/2022.findings-emnlp.108/
%U https://doi.org/10.18653/v1/2022.findings-emnlp.108
%P 1503-1522

Markdown (Informal)

[On the Effectiveness of Automated Metrics for Text Generation Systems](https://aclanthology.org/2022.findings-emnlp.108/) (von Däniken et al., Findings 2022)

• On the Effectiveness of Automated Metrics for Text Generation Systems (von Däniken et al., Findings 2022)

ACL

• Pius von Däniken, Jan Deriu, Don Tuggener, and Mark Cieliebak. 2022. On the Effectiveness of Automated Metrics for Text Generation Systems. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1503–1522, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.