<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="zhang-etal-2024-selective">
    <titleInfo>
        <title>Selective Prefix Tuning for Pre-trained Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Hongyi</namePart>
        <namePart type="family">Zhang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Zuchao</namePart>
        <namePart type="family">Li</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ping</namePart>
        <namePart type="family">Wang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hai</namePart>
        <namePart type="family">Zhao</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-08</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: ACL 2024</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Lun-Wei</namePart>
            <namePart type="family">Ku</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Andre</namePart>
            <namePart type="family">Martins</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Vivek</namePart>
            <namePart type="family">Srikumar</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Bangkok, Thailand</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>The prevalent approach for optimizing pre-trained language models in downstream tasks is fine-tuning. However, it is both time-consuming and memory-inefficient. In response, a more efficient method called Prefix Tuning, which insert learnable vectors into each Transformer layers, has been proposed and proven effective. Recent investigations reveal that prefix tokens carry context-specific information, prompting the hypothesis that enhancing their specialization can improve model performance. To address this, we propose Selective Prefix Tuning (SPT), integrating a selective mechanism inspired by selective self-attention. Additionally, we introduce Selective Loss (SL) to encourage diversity in prefix tokens. Extensive experiments validate the effectiveness of SPT in sentence and token classification tasks. We contribute insight into understanding the role of prefix in model adaptation.</abstract>
    <identifier type="citekey">zhang-etal-2024-selective</identifier>
    <identifier type="doi">10.18653/v1/2024.findings-acl.164</identifier>
    <location>
        <url>https://aclanthology.org/2024.findings-acl.164/</url>
    </location>
    <part>
        <date>2024-08</date>
        <extent unit="page">
            <start>2806</start>
            <end>2813</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Selective Prefix Tuning for Pre-trained Language Models
%A Zhang, Hongyi
%A Li, Zuchao
%A Wang, Ping
%A Zhao, Hai
%Y Ku, Lun-Wei
%Y Martins, Andre
%Y Srikumar, Vivek
%S Findings of the Association for Computational Linguistics: ACL 2024
%D 2024
%8 August
%I Association for Computational Linguistics
%C Bangkok, Thailand
%F zhang-etal-2024-selective
%X The prevalent approach for optimizing pre-trained language models in downstream tasks is fine-tuning. However, it is both time-consuming and memory-inefficient. In response, a more efficient method called Prefix Tuning, which insert learnable vectors into each Transformer layers, has been proposed and proven effective. Recent investigations reveal that prefix tokens carry context-specific information, prompting the hypothesis that enhancing their specialization can improve model performance. To address this, we propose Selective Prefix Tuning (SPT), integrating a selective mechanism inspired by selective self-attention. Additionally, we introduce Selective Loss (SL) to encourage diversity in prefix tokens. Extensive experiments validate the effectiveness of SPT in sentence and token classification tasks. We contribute insight into understanding the role of prefix in model adaptation.
%R 10.18653/v1/2024.findings-acl.164
%U https://aclanthology.org/2024.findings-acl.164/
%U https://doi.org/10.18653/v1/2024.findings-acl.164
%P 2806-2813

Markdown (Informal)

[Selective Prefix Tuning for Pre-trained Language Models](https://aclanthology.org/2024.findings-acl.164/) (Zhang et al., Findings 2024)

• Selective Prefix Tuning for Pre-trained Language Models (Zhang et al., Findings 2024)

ACL

• Hongyi Zhang, Zuchao Li, Ping Wang, and Hai Zhao. 2024. Selective Prefix Tuning for Pre-trained Language Models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2806–2813, Bangkok, Thailand. Association for Computational Linguistics.