Can Wikipedia Categories Improve Masked Language Model Pretraining?

Diksha Meghwal; Katharina von der Wense; Iacer Calixto; Stanislaw Jastrzębski

doi:10.18653/v1/2020.winlp-1.19

Can Wikipedia Categories Improve Masked Language Model Pretraining?

Diksha Meghwal, Katharina Kann, Iacer Calixto, Stanislaw Jastrzebski

Correct Metadata for

Important: The Anthology treat PDFs as authoritative. Please use this form only to correct data that is out of line with the PDF. See our corrections guidelines if you need to change the PDF.

Title Adjust the title. Retain tags such as <fixed-case>.

Authors Adjust author names and order to match the PDF.

Abstract Correct abstract if needed. Retain XML formatting tags such as <tex-math>.

Verification against PDF Ensure that the new title/authors match the snapshot below. (If there is no snapshot or it is too small, consult the PDF.)

Authors concatenated from the text boxes above:

ALL author names match the snapshot above—including middle initials, hyphens, and accents.

Abstract

Pretrained language models have obtained impressive results for a large set of natural language understanding tasks. However, training these models is computationally expensive and requires huge amounts of data. Thus, it would be desirable to automatically detect groups of more or less important examples. Here, we investigate if we can leverage sources of information which are commonly overlooked, Wikipedia categories as listed in DBPedia, to identify useful or harmful data points during pretraining. We define an experimental setup in which we analyze correlations between language model perplexity on specific clusters and downstream NLP task performances during pretraining. Our experiments show that Wikipedia categories are not a good indicator of the importance of specific sentences for pretraining.

Anthology ID:: 2020.winlp-1.19
Volume:: Proceedings of the Fourth Widening Natural Language Processing Workshop
Month:: July
Year:: 2020
Address:: Seattle, USA
Editors:: Rossana Cunha, Samira Shaikh, Erika Varis, Ryan Georgi, Alicia Tsai, Antonios Anastasopoulos, Khyathi Raghavi Chandu
Venue:: WiNLP
SIG:
Publisher:: Association for Computational Linguistics
Note:
Pages:: 78
Language:
URL:: https://aclanthology.org/2020.winlp-1.19/
DOI:: 10.18653/v1/2020.winlp-1.19
Bibkey:
Cite (ACL):: Diksha Meghwal, Katharina Kann, Iacer Calixto, and Stanislaw Jastrzebski. 2020. Can Wikipedia Categories Improve Masked Language Model Pretraining?. In Proceedings of the Fourth Widening Natural Language Processing Workshop, page 78, Seattle, USA. Association for Computational Linguistics.
Cite (Informal):: Can Wikipedia Categories Improve Masked Language Model Pretraining? (Meghwal et al., WiNLP 2020)
Copy Citation:
Video:: http://slideslive.com/38929556

Cite Search Video Fix data

Export citation

BibTeX
MODS XML
Endnote
Preformatted

@inproceedings{meghwal-etal-2020-wikipedia,
    title = "Can {W}ikipedia Categories Improve Masked Language Model Pretraining?",
    author = "Meghwal, Diksha  and
      Kann, Katharina  and
      Calixto, Iacer  and
      Jastrzebski, Stanislaw",
    editor = "Cunha, Rossana  and
      Shaikh, Samira  and
      Varis, Erika  and
      Georgi, Ryan  and
      Tsai, Alicia  and
      Anastasopoulos, Antonios  and
      Chandu, Khyathi Raghavi",
    booktitle = "Proceedings of the Fourth Widening Natural Language Processing Workshop",
    month = jul,
    year = "2020",
    address = "Seattle, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2020.winlp-1.19/",
    doi = "10.18653/v1/2020.winlp-1.19",
    pages = "78",
    abstract = "Pretrained language models have obtained impressive results for a large set of natural language understanding tasks. However, training these models is computationally expensive and requires huge amounts of data. Thus, it would be desirable to automatically detect groups of more or less important examples. Here, we investigate if we can leverage sources of information which are commonly overlooked, Wikipedia categories as listed in DBPedia, to identify useful or harmful data points during pretraining. We define an experimental setup in which we analyze correlations between language model perplexity on specific clusters and downstream NLP task performances during pretraining. Our experiments show that Wikipedia categories are not a good indicator of the importance of specific sentences for pretraining."
}

Download as File

<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="meghwal-etal-2020-wikipedia">
    <titleInfo>
        <title>Can Wikipedia Categories Improve Masked Language Model Pretraining?</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Diksha</namePart>
        <namePart type="family">Meghwal</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Katharina</namePart>
        <namePart type="family">Kann</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Iacer</namePart>
        <namePart type="family">Calixto</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Stanislaw</namePart>
        <namePart type="family">Jastrzebski</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Fourth Widening Natural Language Processing Workshop</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Rossana</namePart>
            <namePart type="family">Cunha</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Samira</namePart>
            <namePart type="family">Shaikh</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Erika</namePart>
            <namePart type="family">Varis</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ryan</namePart>
            <namePart type="family">Georgi</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Alicia</namePart>
            <namePart type="family">Tsai</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Antonios</namePart>
            <namePart type="family">Anastasopoulos</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Khyathi</namePart>
            <namePart type="given">Raghavi</namePart>
            <namePart type="family">Chandu</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Seattle, USA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Pretrained language models have obtained impressive results for a large set of natural language understanding tasks. However, training these models is computationally expensive and requires huge amounts of data. Thus, it would be desirable to automatically detect groups of more or less important examples. Here, we investigate if we can leverage sources of information which are commonly overlooked, Wikipedia categories as listed in DBPedia, to identify useful or harmful data points during pretraining. We define an experimental setup in which we analyze correlations between language model perplexity on specific clusters and downstream NLP task performances during pretraining. Our experiments show that Wikipedia categories are not a good indicator of the importance of specific sentences for pretraining.</abstract>
    <identifier type="citekey">meghwal-etal-2020-wikipedia</identifier>
    <identifier type="doi">10.18653/v1/2020.winlp-1.19</identifier>
    <location>
        <url>https://aclanthology.org/2020.winlp-1.19/</url>
    </location>
    <part>
        <date>2020-07</date>
        <detail type="page"><number>78</number></detail>
    </part>
</mods>
</modsCollection>

Download as File

%0 Conference Proceedings
%T Can Wikipedia Categories Improve Masked Language Model Pretraining?
%A Meghwal, Diksha
%A Kann, Katharina
%A Calixto, Iacer
%A Jastrzebski, Stanislaw
%Y Cunha, Rossana
%Y Shaikh, Samira
%Y Varis, Erika
%Y Georgi, Ryan
%Y Tsai, Alicia
%Y Anastasopoulos, Antonios
%Y Chandu, Khyathi Raghavi
%S Proceedings of the Fourth Widening Natural Language Processing Workshop
%D 2020
%8 July
%I Association for Computational Linguistics
%C Seattle, USA
%F meghwal-etal-2020-wikipedia
%X Pretrained language models have obtained impressive results for a large set of natural language understanding tasks. However, training these models is computationally expensive and requires huge amounts of data. Thus, it would be desirable to automatically detect groups of more or less important examples. Here, we investigate if we can leverage sources of information which are commonly overlooked, Wikipedia categories as listed in DBPedia, to identify useful or harmful data points during pretraining. We define an experimental setup in which we analyze correlations between language model perplexity on specific clusters and downstream NLP task performances during pretraining. Our experiments show that Wikipedia categories are not a good indicator of the importance of specific sentences for pretraining.
%R 10.18653/v1/2020.winlp-1.19
%U https://aclanthology.org/2020.winlp-1.19/
%U https://doi.org/10.18653/v1/2020.winlp-1.19
%P 78

Download as File

Markdown (Informal)

[Can Wikipedia Categories Improve Masked Language Model Pretraining?](https://aclanthology.org/2020.winlp-1.19/) (Meghwal et al., WiNLP 2020)

Can Wikipedia Categories Improve Masked Language Model Pretraining? (Meghwal et al., WiNLP 2020)

ACL

Diksha Meghwal, Katharina Kann, Iacer Calixto, and Stanislaw Jastrzebski. 2020. Can Wikipedia Categories Improve Masked Language Model Pretraining?. In Proceedings of the Fourth Widening Natural Language Processing Workshop, page 78, Seattle, USA. Association for Computational Linguistics.