Towards Evaluation of Cross-document Coreference Resolution Models Using Datasets with Diverse Annotation Schemes

Anastasia Zhukova, Felix Hamborg, Bela Gipp


Abstract
Established cross-document coreference resolution (CDCR) datasets contain event-centric coreference chains of events and entities with identity relations. These datasets establish strict definitions of the coreference relations across related tests but typically ignore anaphora with more vague context-dependent loose coreference relations. In this paper, we qualitatively and quantitatively compare the annotation schemes of ECB+, a CDCR dataset with identity coreference relations, and NewsWCL50, a CDCR dataset with a mix of loose context-dependent and strict coreference relations. We propose a phrasing diversity metric (PD) that encounters for the diversity of full phrases unlike the previously proposed metrics and allows to evaluate lexical diversity of the CDCR datasets in a higher precision. The analysis shows that coreference chains of NewsWCL50 are more lexically diverse than those of ECB+ but annotating of NewsWCL50 leads to the lower inter-coder reliability. We discuss the different tasks that both CDCR datasets create for the CDCR models, i.e., lexical disambiguation and lexical diversity. Finally, to ensure generalizability of the CDCR models, we propose a direction for CDCR evaluation that combines CDCR datasets with multiple annotation schemes that focus of various properties of the coreference chains.
Anthology ID:
2022.lrec-1.522
Volume:
Proceedings of the Thirteenth Language Resources and Evaluation Conference
Month:
June
Year:
2022
Address:
Marseille, France
Editors:
Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
4884–4893
Language:
URL:
https://aclanthology.org/2022.lrec-1.522
DOI:
Bibkey:
Cite (ACL):
Anastasia Zhukova, Felix Hamborg, and Bela Gipp. 2022. Towards Evaluation of Cross-document Coreference Resolution Models Using Datasets with Diverse Annotation Schemes. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4884–4893, Marseille, France. European Language Resources Association.
Cite (Informal):
Towards Evaluation of Cross-document Coreference Resolution Models Using Datasets with Diverse Annotation Schemes (Zhukova et al., LREC 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.lrec-1.522.pdf
Code
 anastasia-zhukova/diverse_cdcr_datasets
Data
ECB+