<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="ranaldi-etal-2023-modeling">
    <titleInfo>
        <title>Modeling Easiness for Training Transformers with Curriculum Learning</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Leonardo</namePart>
        <namePart type="family">Ranaldi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Giulia</namePart>
        <namePart type="family">Pucci</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Fabio</namePart>
        <namePart type="given">Massimo</namePart>
        <namePart type="family">Zanzotto</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-09</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Ruslan</namePart>
            <namePart type="family">Mitkov</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Galia</namePart>
            <namePart type="family">Angelova</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>INCOMA Ltd., Shoumen, Bulgaria</publisher>
            <place>
                <placeTerm type="text">Varna, Bulgaria</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Directly learning from complex examples is generally problematic for humans and machines. Indeed, a better strategy is exposing learners to examples in a reasonable, pedagogically-motivated order. Curriculum Learning (CL) has been proposed to import this strategy when training machine learning models. In this paper, building on Curriculum Learning, we propose a novel, linguistically motivated measure to determine example complexity for organizing examples during learning. Our complexity measure - LRC- is based on length, rarity, and comprehensibility. Our resulting learning model is CL-LRC, that is, CL with LRC. Experiments on downstream tasks show that CL-LRC outperforms existing CL and non-CL methods for training BERT and RoBERTa from scratch. Furthermore, we analyzed different measures, including perplexity, loss, and learning curve of different models pre-trained from scratch, showing that CL-LRC performs better than the state-of-the-art.</abstract>
    <identifier type="citekey">ranaldi-etal-2023-modeling</identifier>
    <location>
        <url>https://aclanthology.org/2023.ranlp-1.101</url>
    </location>
    <part>
        <date>2023-09</date>
        <extent unit="page">
            <start>937</start>
            <end>948</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Modeling Easiness for Training Transformers with Curriculum Learning
%A Ranaldi, Leonardo
%A Pucci, Giulia
%A Zanzotto, Fabio Massimo
%Y Mitkov, Ruslan
%Y Angelova, Galia
%S Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing
%D 2023
%8 September
%I INCOMA Ltd., Shoumen, Bulgaria
%C Varna, Bulgaria
%F ranaldi-etal-2023-modeling
%X Directly learning from complex examples is generally problematic for humans and machines. Indeed, a better strategy is exposing learners to examples in a reasonable, pedagogically-motivated order. Curriculum Learning (CL) has been proposed to import this strategy when training machine learning models. In this paper, building on Curriculum Learning, we propose a novel, linguistically motivated measure to determine example complexity for organizing examples during learning. Our complexity measure - LRC- is based on length, rarity, and comprehensibility. Our resulting learning model is CL-LRC, that is, CL with LRC. Experiments on downstream tasks show that CL-LRC outperforms existing CL and non-CL methods for training BERT and RoBERTa from scratch. Furthermore, we analyzed different measures, including perplexity, loss, and learning curve of different models pre-trained from scratch, showing that CL-LRC performs better than the state-of-the-art.
%U https://aclanthology.org/2023.ranlp-1.101
%P 937-948

Markdown (Informal)

[Modeling Easiness for Training Transformers with Curriculum Learning](https://aclanthology.org/2023.ranlp-1.101) (Ranaldi et al., RANLP 2023)

• Modeling Easiness for Training Transformers with Curriculum Learning (Ranaldi et al., RANLP 2023)

ACL

• Leonardo Ranaldi, Giulia Pucci, and Fabio Massimo Zanzotto. 2023. Modeling Easiness for Training Transformers with Curriculum Learning. In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing, pages 937–948, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria.