2024
pdf
bib
abs
Granular Analysis of Social Media Users’ Truthfulness Stances Toward Climate Change Factual Claims
Haiqi Zhang
|
Zhengyuan Zhu
|
Zeyu Zhang
|
Jacob Devasier
|
Chengkai Li
Proceedings of the 1st Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2024)
Climate change poses an urgent global problem that requires efficient data analysis mechanisms to provide insights into climate-related discussions on social media platforms. This paper presents a framework aimed at understanding social media users’ perceptions of various climate change topics and uncovering the insights behind these perceptions. Our framework employs large language model to develop a taxonomy of factual claims related to climate change and build a classification model that detects the truthfulness stance of tweets toward the factual claims. The findings reveal two key conclusions: (1) The public tends to believe the claims are true, regardless of the actual claim veracity; (2) The public shows a lack of discernment between facts and misinformation across different topics, particularly in areas related to politics, economy, and environment.
pdf
bib
abs
Robust Frame-Semantic Models with Lexical Unit Trees and Negative Samples
Jacob Devasier
|
Yogesh Gurjar
|
Chengkai Li
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
We present novel advancements in frame-semantic parsing, specifically focusing on target identification and frame identification. Our target identification model employs a novel prefix tree modification to enable robust support for multi-word lexical units, resulting in a coverage of 99.4% of the targets in the FrameNet 1.7 fulltext annotations. It utilizes a RoBERTa-based filter to achieve an F1 score of 0.775, surpassing the previous state-of-the-art solution by +0.012. For frame identification, we introduce a modification to the standard multiple-choice classification paradigm by incorporating additional negative frames for targets with limited candidate frames, resulting in a +0.014 accuracy improvement over the frame-only model of FIDO, the previous state-of-the-art system, and +0.002 over its full system. Our approach significantly enhances performance on rare frames, exhibiting an improvement of +0.044 over FIDO’s accuracy on frames with 5 or fewer samples, and on under-utilized frames, with an improvement of +0.139 on targets with a single candidate frame. Overall, our contributions address critical challenges and advance the state-of-the-art in frame-semantic parsing.
2023
pdf
bib
abs
Hallucination Mitigation in Natural Language Generation from Large-Scale Open-Domain Knowledge Graphs
Xiao Shi
|
Zhengyuan Zhu
|
Zeyu Zhang
|
Chengkai Li
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
In generating natural language descriptions for knowledge graph triples, prior works used either small-scale, human-annotated datasets or datasets with limited variety of graph shapes, e.g., those having mostly star graphs. Graph-to-text models trained and evaluated on such datasets are largely not assessed for more realistic large-scale, open-domain settings. We introduce a new dataset, GraphNarrative, to fill this gap. Fine-tuning transformer-based pre-trained language models has achieved state-of-the-art performance among graph-to-text models. However, this method suffers from information hallucination—the generated text may contain fabricated facts not present in input graphs. We propose a novel approach that, given a graph-sentence pair in GraphNarrative, trims the sentence to eliminate portions that are not present in the corresponding graph, by utilizing the sentence’s dependency parse tree. Our experiment results verify this approach using models trained on GraphNarrative and existing datasets. The dataset, source code, and trained models are released at https://github.com/idirlab/graphnarrator.
2021
pdf
bib
abs
A Dashboard for Mitigating the COVID-19 Misinfodemic
Zhengyuan Zhu
|
Kevin Meng
|
Josue Caraballo
|
Israa Jaradat
|
Xiao Shi
|
Zeyu Zhang
|
Farahnaz Akrami
|
Haojin Liao
|
Fatma Arslan
|
Damian Jimenez
|
Mohanmmed Samiul Saeef
|
Paras Pathak
|
Chengkai Li
Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations
This paper describes the current milestones achieved in our ongoing project that aims to understand the surveillance of, impact of and intervention on COVID-19 misinfodemic on Twitter. Specifically, it introduces a public dashboard which, in addition to displaying case counts in an interactive map and a navigational panel, also provides some unique features not found in other places. Particularly, the dashboard uses a curated catalog of COVID-19 related facts and debunks of misinformation, and it displays the most prevalent information from the catalog among Twitter users in user-selected U.S. geographic regions. The paper explains how to use BERT models to match tweets with the facts and misinformation and to detect their stance towards such information. The paper also discusses the results of preliminary experiments on analyzing the spatio-temporal spread of misinformation.
2020
pdf
bib
abs
Modeling Factual Claims with Semantic Frames
Fatma Arslan
|
Josue Caraballo
|
Damian Jimenez
|
Chengkai Li
Proceedings of the Twelfth Language Resources and Evaluation Conference
In this paper, we introduce an extension of the Berkeley FrameNet for the structured and semantic modeling of factual claims. Modeling is a robust tool that can be leveraged in many different tasks such as matching claims to existing fact-checks and translating claims to structured queries. Our work introduces 11 new manually crafted frames along with 9 existing FrameNet frames, all of which have been selected with fact-checking in mind. Along with these frames, we are also providing 2,540 fully annotated sentences, which can be used to understand how these frames are intended to work and to train machine learning models. Finally, we are also releasing our annotation tool to facilitate other researchers to make their own local extensions to FrameNet.
pdf
bib
abs
Jennifer for COVID-19: An NLP-Powered Chatbot Built for the People and by the People to Combat Misinformation
Yunyao Li
|
Tyrone Grandison
|
Patricia Silveyra
|
Ali Douraghy
|
Xinyu Guan
|
Thomas Kieselbach
|
Chengkai Li
|
Haiqi Zhang
Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020
Just as SARS-CoV-2, a new form of coronavirus continues to infect a growing number of people around the world, harmful misinformation about the outbreak also continues to spread. With the goal of combating misinformation, we designed and built Jennifer–a chatbot maintained by a global group of volunteers. With Jennifer, we hope to learn whether public information from reputable sources could be more effectively organized and shared in the wake of a crisis as well as to understand issues that the public were most immediately curious about. In this paper, we introduce Jennifer and describe the design of this proof-of-principle system. We also present lessons learned and discuss open challenges. Finally, to facilitate future research, we release COVID-19 Question Bank, a dataset of 3,924 COVID-19-related questions in 944 groups, gathered from our users and volunteers.
2019
pdf
bib
abs
ClaimPortal: Integrated Monitoring, Searching, Checking, and Analytics of Factual Claims on Twitter
Sarthak Majithia
|
Fatma Arslan
|
Sumeet Lubal
|
Damian Jimenez
|
Priyank Arora
|
Josue Caraballo
|
Chengkai Li
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations
We present ClaimPortal, a web-based platform for monitoring, searching, checking, and analyzing English factual claims on Twitter from the American political domain. We explain the architecture of ClaimPortal, its components and functions, and the user interface. While the last several years have witnessed a substantial growth in interests and efforts in the area of computational fact-checking, ClaimPortal is a novel infrastructure in that fact-checkers have largely skipped factual claims in tweets. It can be a highly powerful tool to both general web users and fact-checkers. It will also be an educational resource in helping cultivate a society that is less susceptible to falsehoods.