Understanding Fine-grained Distortions in Reports of Scientific Findings

Amelie Wuehrl, Dustin Wright, Roman Klinger, Isabelle Augenstein


Abstract
Distorted science communication harms individuals and society as it can lead to unhealthy behavior change and decrease trust in scientific institutions. Given the rapidly increasing volume of science communication in recent years, a fine-grained understanding of how findings from scientific publications are reported to the general public, and methods to detect distortions from the original work automatically, are crucial. Prior work focused on individual aspects of distortions or worked with unpaired data. In this work, we make three foundational contributions towards addressing this problem: (1) annotating 1,600 instances of scientific findings from academic papers paired with corresponding findings as reported in news articles and tweets wrt. four characteristics: causality, certainty, generality and sensationalism; (2) establishing baselines for automatically detecting these characteristics; and (3) analyzing the prevalence of changes in these characteristics in both human-annotated and large-scale unlabeled data. Our results show that scientific findings frequently undergo subtle distortions when reported. Tweets distort findings more often than science news reports. Detecting fine-grained distortions automatically poses a challenging task. In our experiments, fine-tuned task-specific models consistently outperform few-shot LLM prompting.
Anthology ID:
2024.findings-acl.369
Volume:
Findings of the Association for Computational Linguistics ACL 2024
Month:
August
Year:
2024
Address:
Bangkok, Thailand and virtual meeting
Editors:
Lun-Wei Ku, Andre Martins, Vivek Srikumar
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
6175–6191
Language:
URL:
https://aclanthology.org/2024.findings-acl.369
DOI:
10.18653/v1/2024.findings-acl.369
Bibkey:
Cite (ACL):
Amelie Wuehrl, Dustin Wright, Roman Klinger, and Isabelle Augenstein. 2024. Understanding Fine-grained Distortions in Reports of Scientific Findings. In Findings of the Association for Computational Linguistics ACL 2024, pages 6175–6191, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
Cite (Informal):
Understanding Fine-grained Distortions in Reports of Scientific Findings (Wuehrl et al., Findings 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.findings-acl.369.pdf