<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="huber-carenini-2022-towards">
    <titleInfo>
        <title>Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Patrick</namePart>
        <namePart type="family">Huber</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Giuseppe</namePart>
        <namePart type="family">Carenini</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Marine</namePart>
            <namePart type="family">Carpuat</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Marie-Catherine</namePart>
            <namePart type="family">de Marneffe</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ivan</namePart>
            <namePart type="given">Vladimir</namePart>
            <namePart type="family">Meza Ruiz</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Seattle, United States</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>In this paper, we extend the line of BERTology work by focusing on the important, yet less explored, alignment of pre-trained and fine-tuned PLMs with large-scale discourse structures. We propose a novel approach to infer discourse information for arbitrarily long documents. In our experiments, we find that the captured discourse information is local and general, even across a collection of fine-tuning tasks. We compare the inferred discourse trees with supervised, distantly supervised and simple baselines to explore the structural overlap, finding that constituency discourse trees align well with supervised models, however, contain complementary discourse information. Lastly, we individually explore self-attention matrices to analyze the information redundancy. We find that similar discourse information is consistently captured in the same heads.</abstract>
    <identifier type="citekey">huber-carenini-2022-towards</identifier>
    <identifier type="doi">10.18653/v1/2022.naacl-main.170</identifier>
    <location>
        <url>https://aclanthology.org/2022.naacl-main.170/</url>
    </location>
    <part>
        <date>2022-07</date>
        <extent unit="page">
            <start>2376</start>
            <end>2394</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models
%A Huber, Patrick
%A Carenini, Giuseppe
%Y Carpuat, Marine
%Y de Marneffe, Marie-Catherine
%Y Meza Ruiz, Ivan Vladimir
%S Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
%D 2022
%8 July
%I Association for Computational Linguistics
%C Seattle, United States
%F huber-carenini-2022-towards
%X In this paper, we extend the line of BERTology work by focusing on the important, yet less explored, alignment of pre-trained and fine-tuned PLMs with large-scale discourse structures. We propose a novel approach to infer discourse information for arbitrarily long documents. In our experiments, we find that the captured discourse information is local and general, even across a collection of fine-tuning tasks. We compare the inferred discourse trees with supervised, distantly supervised and simple baselines to explore the structural overlap, finding that constituency discourse trees align well with supervised models, however, contain complementary discourse information. Lastly, we individually explore self-attention matrices to analyze the information redundancy. We find that similar discourse information is consistently captured in the same heads.
%R 10.18653/v1/2022.naacl-main.170
%U https://aclanthology.org/2022.naacl-main.170/
%U https://doi.org/10.18653/v1/2022.naacl-main.170
%P 2376-2394

Markdown (Informal)

[Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models](https://aclanthology.org/2022.naacl-main.170/) (Huber & Carenini, NAACL 2022)

• Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models (Huber & Carenini, NAACL 2022)

ACL

• Patrick Huber and Giuseppe Carenini. 2022. Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2376–2394, Seattle, United States. Association for Computational Linguistics.