<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="cattan-etal-2021-realistic">
    <titleInfo>
        <title>Realistic Evaluation Principles for Cross-document Coreference Resolution</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Arie</namePart>
        <namePart type="family">Cattan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Alon</namePart>
        <namePart type="family">Eirew</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Gabriel</namePart>
        <namePart type="family">Stanovsky</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Mandar</namePart>
        <namePart type="family">Joshi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ido</namePart>
        <namePart type="family">Dagan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-08</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Lun-Wei</namePart>
            <namePart type="family">Ku</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Vivi</namePart>
            <namePart type="family">Nastase</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ivan</namePart>
            <namePart type="family">Vulić</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>We point out that common evaluation practices for cross-document coreference resolution have been unrealistically permissive in their assumed settings, yielding inflated results. We propose addressing this issue via two evaluation methodology principles. First, as in other tasks, models should be evaluated on predicted mentions rather than on gold mentions. Doing this raises a subtle issue regarding singleton coreference clusters, which we address by decoupling the evaluation of mention detection from that of coreference linking. Second, we argue that models should not exploit the synthetic topic structure of the standard ECB+ dataset, forcing models to confront the lexical ambiguity challenge, as intended by the dataset creators. We demonstrate empirically the drastic impact of our more realistic evaluation principles on a competitive model, yielding a score which is 33 F1 lower compared to evaluating by prior lenient practices.</abstract>
    <identifier type="citekey">cattan-etal-2021-realistic</identifier>
    <identifier type="doi">10.18653/v1/2021.starsem-1.13</identifier>
    <location>
        <url>https://aclanthology.org/2021.starsem-1.13/</url>
    </location>
    <part>
        <date>2021-08</date>
        <extent unit="page">
            <start>143</start>
            <end>151</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Realistic Evaluation Principles for Cross-document Coreference Resolution
%A Cattan, Arie
%A Eirew, Alon
%A Stanovsky, Gabriel
%A Joshi, Mandar
%A Dagan, Ido
%Y Ku, Lun-Wei
%Y Nastase, Vivi
%Y Vulić, Ivan
%S Proceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics
%D 2021
%8 August
%I Association for Computational Linguistics
%C Online
%F cattan-etal-2021-realistic
%X We point out that common evaluation practices for cross-document coreference resolution have been unrealistically permissive in their assumed settings, yielding inflated results. We propose addressing this issue via two evaluation methodology principles. First, as in other tasks, models should be evaluated on predicted mentions rather than on gold mentions. Doing this raises a subtle issue regarding singleton coreference clusters, which we address by decoupling the evaluation of mention detection from that of coreference linking. Second, we argue that models should not exploit the synthetic topic structure of the standard ECB+ dataset, forcing models to confront the lexical ambiguity challenge, as intended by the dataset creators. We demonstrate empirically the drastic impact of our more realistic evaluation principles on a competitive model, yielding a score which is 33 F1 lower compared to evaluating by prior lenient practices.
%R 10.18653/v1/2021.starsem-1.13
%U https://aclanthology.org/2021.starsem-1.13/
%U https://doi.org/10.18653/v1/2021.starsem-1.13
%P 143-151

Markdown (Informal)

[Realistic Evaluation Principles for Cross-document Coreference Resolution](https://aclanthology.org/2021.starsem-1.13/) (Cattan et al., *SEM 2021)

• Realistic Evaluation Principles for Cross-document Coreference Resolution (Cattan et al., *SEM 2021)

ACL

• Arie Cattan, Alon Eirew, Gabriel Stanovsky, Mandar Joshi, and Ido Dagan. 2021. Realistic Evaluation Principles for Cross-document Coreference Resolution. In Proceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics, pages 143–151, Online. Association for Computational Linguistics.