<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="dhingra-etal-2022-time">
    <titleInfo>
        <title>Time-Aware Language Models as Temporal Knowledge Bases</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Bhuwan</namePart>
        <namePart type="family">Dhingra</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jeremy</namePart>
        <namePart type="given">R</namePart>
        <namePart type="family">Cole</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Julian</namePart>
        <namePart type="given">Martin</namePart>
        <namePart type="family">Eisenschlos</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Daniel</namePart>
        <namePart type="family">Gillick</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jacob</namePart>
        <namePart type="family">Eisenstein</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">William</namePart>
        <namePart type="given">W</namePart>
        <namePart type="family">Cohen</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Many facts come with an expiration date, from the name of the President to the basketball team Lebron James plays for. However, most language models (LMs) are trained on snapshots of data collected at a specific moment in time. This can limit their utility, especially in the closed-book setting where the pretraining corpus must contain the facts the model should memorize. We introduce a diagnostic dataset aimed at probing LMs for factual knowledge that changes over time and highlight problems with LMs at either end of the spectrum—those trained on specific slices of temporal data, as well as those trained on a wide range of temporal data. To mitigate these problems, we propose a simple technique for jointly modeling text with its timestamp. This improves memorization of seen facts from the training time period, as well as calibration on predictions about unseen facts from future time periods. We also show that models trained with temporal context can be efficiently “refreshed” as new data arrives, without the need for retraining from scratch.</abstract>
    <identifier type="citekey">dhingra-etal-2022-time</identifier>
    <identifier type="doi">10.1162/tacl_a_00459</identifier>
    <location>
        <url>https://aclanthology.org/2022.tacl-1.15</url>
    </location>
    <part>
        <date>2022</date>
        <detail type="volume"><number>10</number></detail>
        <extent unit="page">
            <start>257</start>
            <end>273</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T Time-Aware Language Models as Temporal Knowledge Bases
%A Dhingra, Bhuwan
%A Cole, Jeremy R.
%A Eisenschlos, Julian Martin
%A Gillick, Daniel
%A Eisenstein, Jacob
%A Cohen, William W.
%J Transactions of the Association for Computational Linguistics
%D 2022
%V 10
%I MIT Press
%C Cambridge, MA
%F dhingra-etal-2022-time
%X Many facts come with an expiration date, from the name of the President to the basketball team Lebron James plays for. However, most language models (LMs) are trained on snapshots of data collected at a specific moment in time. This can limit their utility, especially in the closed-book setting where the pretraining corpus must contain the facts the model should memorize. We introduce a diagnostic dataset aimed at probing LMs for factual knowledge that changes over time and highlight problems with LMs at either end of the spectrum—those trained on specific slices of temporal data, as well as those trained on a wide range of temporal data. To mitigate these problems, we propose a simple technique for jointly modeling text with its timestamp. This improves memorization of seen facts from the training time period, as well as calibration on predictions about unseen facts from future time periods. We also show that models trained with temporal context can be efficiently “refreshed” as new data arrives, without the need for retraining from scratch.
%R 10.1162/tacl_a_00459
%U https://aclanthology.org/2022.tacl-1.15
%U https://doi.org/10.1162/tacl_a_00459
%P 257-273

Markdown (Informal)

[Time-Aware Language Models as Temporal Knowledge Bases](https://aclanthology.org/2022.tacl-1.15) (Dhingra et al., TACL 2022)

• Time-Aware Language Models as Temporal Knowledge Bases (Dhingra et al., TACL 2022)

ACL

• Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022. Time-Aware Language Models as Temporal Knowledge Bases. Transactions of the Association for Computational Linguistics, 10:257–273.