<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="zhu-tan-2023-spt">
    <titleInfo>
        <title>SPT: Learning to Selectively Insert Prompts for Better Prompt Tuning</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Wei</namePart>
        <namePart type="family">Zhu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ming</namePart>
        <namePart type="family">Tan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Prompt tuning prepends a soft prompt to the input embeddings or hidden states and only optimizes the prompt to adapt pretrained models (PTMs) to downstream tasks. The previous work manually selects prompt layers which are far from optimal and failed to exploit the potential of prompt tuning. In this work, we propose a novel framework, Selective Prompt Tuning (SPT), that learns to select the proper prompt layers by inserting a prompt controlled by a learnable probabilistic gate at each intermediate layer. We further propose a novel bi-level optimization framework, SPT-DARTS, that can better optimize the learnable gates and improve the final prompt tuning performances of the learned prompt layer settings. We conduct extensive experiments with ten benchmark datasets under the full-data and few-shot scenarios. The results demonstrate that our SPT framework can perform better than the previous state-of-the-art PETuning baselines with comparable or fewer tunable parameters.</abstract>
    <identifier type="citekey">zhu-tan-2023-spt</identifier>
    <identifier type="doi">10.18653/v1/2023.emnlp-main.727</identifier>
    <location>
        <url>https://aclanthology.org/2023.emnlp-main.727</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>11862</start>
            <end>11878</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T SPT: Learning to Selectively Insert Prompts for Better Prompt Tuning
%A Zhu, Wei
%A Tan, Ming
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F zhu-tan-2023-spt
%X Prompt tuning prepends a soft prompt to the input embeddings or hidden states and only optimizes the prompt to adapt pretrained models (PTMs) to downstream tasks. The previous work manually selects prompt layers which are far from optimal and failed to exploit the potential of prompt tuning. In this work, we propose a novel framework, Selective Prompt Tuning (SPT), that learns to select the proper prompt layers by inserting a prompt controlled by a learnable probabilistic gate at each intermediate layer. We further propose a novel bi-level optimization framework, SPT-DARTS, that can better optimize the learnable gates and improve the final prompt tuning performances of the learned prompt layer settings. We conduct extensive experiments with ten benchmark datasets under the full-data and few-shot scenarios. The results demonstrate that our SPT framework can perform better than the previous state-of-the-art PETuning baselines with comparable or fewer tunable parameters.
%R 10.18653/v1/2023.emnlp-main.727
%U https://aclanthology.org/2023.emnlp-main.727
%U https://doi.org/10.18653/v1/2023.emnlp-main.727
%P 11862-11878

Markdown (Informal)

[SPT: Learning to Selectively Insert Prompts for Better Prompt Tuning](https://aclanthology.org/2023.emnlp-main.727) (Zhu & Tan, EMNLP 2023)

• SPT: Learning to Selectively Insert Prompts for Better Prompt Tuning (Zhu & Tan, EMNLP 2023)

ACL

• Wei Zhu and Ming Tan. 2023. SPT: Learning to Selectively Insert Prompts for Better Prompt Tuning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11862–11878, Singapore. Association for Computational Linguistics.