A Benchmark Corpus for the Detection of Automatically Generated Text in Academic Publications

Vijini Liyanage, Davide Buscaldi, Adeline Nazarenko


Abstract
Automatic text generation based on neural language models has achieved performance levels that make the generated text almost indistinguishable from those written by humans. Despite the value that text generation can have in various applications, it can also be employed for malicious tasks. The diffusion of such practices represent a threat to the quality of academic publishing. To address these problems, we propose in this paper two datasets comprised of artificially generated research content: a completely synthetic dataset and a partial text substitution dataset. In the first case, the content is completely generated by the GPT-2 model after a short prompt extracted from original papers. The partial or hybrid dataset is created by replacing several sentences of abstracts with sentences that are generated by the Arxiv-NLP model. We evaluate the quality of the datasets comparing the generated texts to aligned original texts using fluency metrics such as BLEU and ROUGE. The more natural the artificial texts seem, the more difficult they are to detect and the better is the benchmark. We also evaluate the difficulty of the task of distinguishing original from generated text by using state-of-the-art classification models.
Anthology ID:
2022.lrec-1.501
Volume:
Proceedings of the Thirteenth Language Resources and Evaluation Conference
Month:
June
Year:
2022
Address:
Marseille, France
Editors:
Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
4692–4700
Language:
URL:
https://aclanthology.org/2022.lrec-1.501
DOI:
Bibkey:
Cite (ACL):
Vijini Liyanage, Davide Buscaldi, and Adeline Nazarenko. 2022. A Benchmark Corpus for the Detection of Automatically Generated Text in Academic Publications. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4692–4700, Marseille, France. European Language Resources Association.
Cite (Informal):
A Benchmark Corpus for the Detection of Automatically Generated Text in Academic Publications (Liyanage et al., LREC 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.lrec-1.501.pdf
Code
 vijini/generatedtextdetection