Evaluating Discourse Cohesion in Pre-trained Language Models

Jie He, Wanqiu Long, Deyi Xiong


Abstract
Large pre-trained neural models have achieved remarkable success in natural language process (NLP), inspiring a growing body of research analyzing their ability from different aspects. In this paper, we propose a test suite to evaluate the cohesive ability of pre-trained language models. The test suite contains multiple cohesion phenomena between adjacent and non-adjacent sentences. We try to compare different pre-trained language models on these phenomena and analyze the experimental results,hoping more attention can be given to discourse cohesion in the future. The built discourse cohesion test suite will be publicly available at https://github.com/probe2/discourse_cohesion.
Anthology ID:
2022.codi-1.4
Volume:
Proceedings of the 3rd Workshop on Computational Approaches to Discourse
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea and Online
Editors:
Chloe Braud, Christian Hardmeier, Junyi Jessy Li, Sharid Loaiciga, Michael Strube, Amir Zeldes
Venue:
CODI
SIG:
Publisher:
International Conference on Computational Linguistics
Note:
Pages:
28–34
Language:
URL:
https://aclanthology.org/2022.codi-1.4
DOI:
Bibkey:
Cite (ACL):
Jie He, Wanqiu Long, and Deyi Xiong. 2022. Evaluating Discourse Cohesion in Pre-trained Language Models. In Proceedings of the 3rd Workshop on Computational Approaches to Discourse, pages 28–34, Gyeongju, Republic of Korea and Online. International Conference on Computational Linguistics.
Cite (Informal):
Evaluating Discourse Cohesion in Pre-trained Language Models (He et al., CODI 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.codi-1.4.pdf