On the Robustness of Document-Level Relation Extraction Models to Entity Name Variations

Shiao Meng, Xuming Hu, Aiwei Liu, Fukun Ma, Yawen Yang, Shuang Li, Lijie Wen


Abstract
Driven by the demand for cross-sentence and large-scale relation extraction, document-level relation extraction (DocRE) has attracted increasing research interest. Despite the continuous improvement in performance, we find that existing DocRE models which initially perform well may make more mistakes when merely changing the entity names in the document, hindering the generalization to novel entity names. To this end, we systematically investigate the robustness of DocRE models to entity name variations in this work. We first propose a principled pipeline to generate entity-renamed documents by replacing the original entity names with names from Wikidata. By applying the pipeline to DocRED and Re-DocRED datasets, we construct two novel benchmarks named Env-DocRED and Env-Re-DocRED for robustness evaluation. Experimental results show that both three representative DocRE models and two in-context learned large language models consistently lack sufficient robustness to entity name variations, particularly on cross-sentence relation instances and documents with more entities. Finally, we propose an entity variation robust training method which not only improves the robustness of DocRE models but also enhances their understanding and reasoning capabilities. We further verify that the basic idea of this method can be effectively transferred to in-context learning for DocRE as well.
Anthology ID:
2024.findings-acl.969
Volume:
Findings of the Association for Computational Linguistics ACL 2024
Month:
August
Year:
2024
Address:
Bangkok, Thailand and virtual meeting
Editors:
Lun-Wei Ku, Andre Martins, Vivek Srikumar
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
16362–16374
Language:
URL:
https://aclanthology.org/2024.findings-acl.969
DOI:
10.18653/v1/2024.findings-acl.969
Bibkey:
Cite (ACL):
Shiao Meng, Xuming Hu, Aiwei Liu, Fukun Ma, Yawen Yang, Shuang Li, and Lijie Wen. 2024. On the Robustness of Document-Level Relation Extraction Models to Entity Name Variations. In Findings of the Association for Computational Linguistics ACL 2024, pages 16362–16374, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
Cite (Informal):
On the Robustness of Document-Level Relation Extraction Models to Entity Name Variations (Meng et al., Findings 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.findings-acl.969.pdf