<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="miletic-walde-2024-semantics">
    <titleInfo>
        <title>Semantics of Multiword Expressions in Transformer-Based Models: A Survey</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Filip</namePart>
        <namePart type="family">Miletić</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Sabine</namePart>
        <namePart type="given">Schulte</namePart>
        <namePart type="given">im</namePart>
        <namePart type="family">Walde</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Multiword expressions (MWEs) are composed of multiple words and exhibit variable degrees of compositionality. As such, their meanings are notoriously difficult to model, and it is unclear to what extent this issue affects transformer architectures. Addressing this gap, we provide the first in-depth survey of MWE processing with transformer models. We overall find that they capture MWE semantics inconsistently, as shown by reliance on surface patterns and memorized information. MWE meaning is also strongly localized, predominantly in early layers of the architecture. Representations benefit from specific linguistic properties, such as lower semantic idiosyncrasy and ambiguity of target expressions. Our findings overall question the ability of transformer models to robustly capture fine-grained semantics. Furthermore, we highlight the need for more directly comparable evaluation setups.</abstract>
    <identifier type="citekey">miletic-walde-2024-semantics</identifier>
    <identifier type="doi">10.1162/tacl_a_00657</identifier>
    <location>
        <url>https://aclanthology.org/2024.tacl-1.33/</url>
    </location>
    <part>
        <date>2024</date>
        <detail type="volume"><number>12</number></detail>
        <extent unit="page">
            <start>593</start>
            <end>612</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T Semantics of Multiword Expressions in Transformer-Based Models: A Survey
%A Miletić, Filip
%A Walde, Sabine Schulte im
%J Transactions of the Association for Computational Linguistics
%D 2024
%V 12
%I MIT Press
%C Cambridge, MA
%F miletic-walde-2024-semantics
%X Multiword expressions (MWEs) are composed of multiple words and exhibit variable degrees of compositionality. As such, their meanings are notoriously difficult to model, and it is unclear to what extent this issue affects transformer architectures. Addressing this gap, we provide the first in-depth survey of MWE processing with transformer models. We overall find that they capture MWE semantics inconsistently, as shown by reliance on surface patterns and memorized information. MWE meaning is also strongly localized, predominantly in early layers of the architecture. Representations benefit from specific linguistic properties, such as lower semantic idiosyncrasy and ambiguity of target expressions. Our findings overall question the ability of transformer models to robustly capture fine-grained semantics. Furthermore, we highlight the need for more directly comparable evaluation setups.
%R 10.1162/tacl_a_00657
%U https://aclanthology.org/2024.tacl-1.33/
%U https://doi.org/10.1162/tacl_a_00657
%P 593-612

Markdown (Informal)

[Semantics of Multiword Expressions in Transformer-Based Models: A Survey](https://aclanthology.org/2024.tacl-1.33/) (Miletić & Walde, TACL 2024)

• Semantics of Multiword Expressions in Transformer-Based Models: A Survey (Miletić & Walde, TACL 2024)

ACL

• Filip Miletić and Sabine Schulte im Walde. 2024. Semantics of Multiword Expressions in Transformer-Based Models: A Survey. Transactions of the Association for Computational Linguistics, 12:593–612.