<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="stepanova-ross-2023-temporal">
    <titleInfo>
        <title>Temporal Generalizability in Multimodal Misinformation Detection</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Nataliya</namePart>
        <namePart type="family">Stepanova</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Björn</namePart>
        <namePart type="family">Ross</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Dieuwke</namePart>
            <namePart type="family">Hupkes</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Verna</namePart>
            <namePart type="family">Dankers</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Khuyagbaatar</namePart>
            <namePart type="family">Batsuren</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Koustuv</namePart>
            <namePart type="family">Sinha</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Amirhossein</namePart>
            <namePart type="family">Kazemnejad</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Christos</namePart>
            <namePart type="family">Christodoulopoulos</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ryan</namePart>
            <namePart type="family">Cotterell</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Elia</namePart>
            <namePart type="family">Bruni</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Misinformation detection models degrade in performance over time, but the precise causes of this remain under-researched, in particular for multimodal models. We present experiments investigating the impact of temporal shift on performance of multimodal automatic misinformation detection classifiers. Working with the r/Fakeddit dataset, we found that evaluating models on temporally out-of-domain data (i.e. data from time stretches unseen in training) results in a non-linear, 7-8% drop in macro F1 as compared to traditional evaluation strategies (which do not control for the effect of content change over time). Focusing on two factors that make temporal generalizability in misinformation detection difficult, content shift and class distribution shift, we found that content shift has a stronger effect on recall. Within the context of coarse-grained vs. fine-grained misinformation detection with r/Fakeddit, we find that certain misinformation classes seem to be more stable with respect to content shift (e.g. Manipulated and Misleading Content). Our results indicate that future research efforts need to explicitly account for the temporal nature of misinformation to ensure that experiments reflect expected real-world performance.</abstract>
    <identifier type="citekey">stepanova-ross-2023-temporal</identifier>
    <identifier type="doi">10.18653/v1/2023.genbench-1.6</identifier>
    <location>
        <url>https://aclanthology.org/2023.genbench-1.6/</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>76</start>
            <end>88</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Temporal Generalizability in Multimodal Misinformation Detection
%A Stepanova, Nataliya
%A Ross, Björn
%Y Hupkes, Dieuwke
%Y Dankers, Verna
%Y Batsuren, Khuyagbaatar
%Y Sinha, Koustuv
%Y Kazemnejad, Amirhossein
%Y Christodoulopoulos, Christos
%Y Cotterell, Ryan
%Y Bruni, Elia
%S Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F stepanova-ross-2023-temporal
%X Misinformation detection models degrade in performance over time, but the precise causes of this remain under-researched, in particular for multimodal models. We present experiments investigating the impact of temporal shift on performance of multimodal automatic misinformation detection classifiers. Working with the r/Fakeddit dataset, we found that evaluating models on temporally out-of-domain data (i.e. data from time stretches unseen in training) results in a non-linear, 7-8% drop in macro F1 as compared to traditional evaluation strategies (which do not control for the effect of content change over time). Focusing on two factors that make temporal generalizability in misinformation detection difficult, content shift and class distribution shift, we found that content shift has a stronger effect on recall. Within the context of coarse-grained vs. fine-grained misinformation detection with r/Fakeddit, we find that certain misinformation classes seem to be more stable with respect to content shift (e.g. Manipulated and Misleading Content). Our results indicate that future research efforts need to explicitly account for the temporal nature of misinformation to ensure that experiments reflect expected real-world performance.
%R 10.18653/v1/2023.genbench-1.6
%U https://aclanthology.org/2023.genbench-1.6/
%U https://doi.org/10.18653/v1/2023.genbench-1.6
%P 76-88

Markdown (Informal)

[Temporal Generalizability in Multimodal Misinformation Detection](https://aclanthology.org/2023.genbench-1.6/) (Stepanova & Ross, GenBench 2023)

• Temporal Generalizability in Multimodal Misinformation Detection (Stepanova & Ross, GenBench 2023)

ACL

• Nataliya Stepanova and Björn Ross. 2023. Temporal Generalizability in Multimodal Misinformation Detection. In Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP, pages 76–88, Singapore. Association for Computational Linguistics.