Quantifying Contextual Aspects of Inter-annotator Agreement in Intertextuality Research

Enrique Manjavacas Arevalo, Laurence Mellerin, Mike Kestemont


Abstract
We report on an inter-annotator agreement experiment involving instances of text reuse focusing on the well-known case of biblical intertextuality in medieval literature. We target the application use case of literary scholars whose aim is to document instances of biblical references in the ‘apparatus fontium’ of a prospective digital edition. We develop a Bayesian implementation of Cohen’s kappa for multiple annotators that allows us to assess the influence of various contextual effects on the inter-annotator agreement, producing both more robust estimates of the agreement indices as well as insights into the annotation process that leads to the estimated indices. As a result, we are able to produce a novel and nuanced estimation of inter-annotator agreement in the context of intertextuality, exploring the challenges that arise from manually annotating a dataset of biblical references in the writings of Bernard of Clairvaux. Among others, our method was able to unveil the fact that the obtained agreement depends heavily on the biblical source book of the proposed reference, as well as the underlying algorithm used to retrieve the candidate match.
Anthology ID:
2021.latechclfl-1.4
Volume:
Proceedings of the 5th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature
Month:
November
Year:
2021
Address:
Punta Cana, Dominican Republic (online)
Editors:
Stefania Degaetano-Ortlieb, Anna Kazantseva, Nils Reiter, Stan Szpakowicz
Venue:
LaTeCHCLfL
SIG:
SIGHUM
Publisher:
Association for Computational Linguistics
Note:
Pages:
31–42
Language:
URL:
https://aclanthology.org/2021.latechclfl-1.4
DOI:
10.18653/v1/2021.latechclfl-1.4
Bibkey:
Cite (ACL):
Enrique Manjavacas Arevalo, Laurence Mellerin, and Mike Kestemont. 2021. Quantifying Contextual Aspects of Inter-annotator Agreement in Intertextuality Research. In Proceedings of the 5th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pages 31–42, Punta Cana, Dominican Republic (online). Association for Computational Linguistics.
Cite (Informal):
Quantifying Contextual Aspects of Inter-annotator Agreement in Intertextuality Research (Manjavacas Arevalo et al., LaTeCHCLfL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.latechclfl-1.4.pdf
Video:
 https://aclanthology.org/2021.latechclfl-1.4.mp4