<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="kobayashi-etal-2024-revisiting">
    <titleInfo>
        <title>Revisiting Meta-evaluation for Grammatical Error Correction</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Masamune</namePart>
        <namePart type="family">Kobayashi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Masato</namePart>
        <namePart type="family">Mita</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Mamoru</namePart>
        <namePart type="family">Komachi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Metrics are the foundation for automatic evaluation in grammatical error correction (GEC), with their evaluation of the metrics (meta-evaluation) relying on their correlation with human judgments. However, conventional meta-evaluations in English GEC encounter several challenges, including biases caused by inconsistencies in evaluation granularity and an outdated setup using classical systems. These problems can lead to misinterpretation of metrics and potentially hinder the applicability of GEC techniques. To address these issues, this paper proposes SEEDA, a new dataset for GEC meta-evaluation. SEEDA consists of corrections with human ratings along two different granularities: edit-based and sentence-based, covering 12 state-of-the-art systems including large language models, and two human corrections with different focuses. The results of improved correlations by aligning the granularity in the sentence-level meta-evaluation suggest that edit-based metrics may have been underestimated in existing studies. Furthermore, correlations of most metrics decrease when changing from classical to neural systems, indicating that traditional metrics are relatively poor at evaluating fluently corrected sentences with many edits.</abstract>
    <identifier type="citekey">kobayashi-etal-2024-revisiting</identifier>
    <identifier type="doi">10.1162/tacl_a_00676</identifier>
    <location>
        <url>https://aclanthology.org/2024.tacl-1.47/</url>
    </location>
    <part>
        <date>2024</date>
        <detail type="volume"><number>12</number></detail>
        <extent unit="page">
            <start>837</start>
            <end>855</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T Revisiting Meta-evaluation for Grammatical Error Correction
%A Kobayashi, Masamune
%A Mita, Masato
%A Komachi, Mamoru
%J Transactions of the Association for Computational Linguistics
%D 2024
%V 12
%I MIT Press
%C Cambridge, MA
%F kobayashi-etal-2024-revisiting
%X Metrics are the foundation for automatic evaluation in grammatical error correction (GEC), with their evaluation of the metrics (meta-evaluation) relying on their correlation with human judgments. However, conventional meta-evaluations in English GEC encounter several challenges, including biases caused by inconsistencies in evaluation granularity and an outdated setup using classical systems. These problems can lead to misinterpretation of metrics and potentially hinder the applicability of GEC techniques. To address these issues, this paper proposes SEEDA, a new dataset for GEC meta-evaluation. SEEDA consists of corrections with human ratings along two different granularities: edit-based and sentence-based, covering 12 state-of-the-art systems including large language models, and two human corrections with different focuses. The results of improved correlations by aligning the granularity in the sentence-level meta-evaluation suggest that edit-based metrics may have been underestimated in existing studies. Furthermore, correlations of most metrics decrease when changing from classical to neural systems, indicating that traditional metrics are relatively poor at evaluating fluently corrected sentences with many edits.
%R 10.1162/tacl_a_00676
%U https://aclanthology.org/2024.tacl-1.47/
%U https://doi.org/10.1162/tacl_a_00676
%P 837-855

Markdown (Informal)

[Revisiting Meta-evaluation for Grammatical Error Correction](https://aclanthology.org/2024.tacl-1.47/) (Kobayashi et al., TACL 2024)

• Revisiting Meta-evaluation for Grammatical Error Correction (Kobayashi et al., TACL 2024)

ACL

• Masamune Kobayashi, Masato Mita, and Mamoru Komachi. 2024. Revisiting Meta-evaluation for Grammatical Error Correction. Transactions of the Association for Computational Linguistics, 12:837–855.