<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="ye-durrett-2023-explanation">
    <titleInfo>
        <title>Explanation Selection Using Unlabeled Data for Chain-of-Thought Prompting</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Xi</namePart>
        <namePart type="family">Ye</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Greg</namePart>
        <namePart type="family">Durrett</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Recent work has shown how to prompt large language models with explanations to obtain strong performance on textual reasoning tasks, i.e., the chain-of-thought paradigm. However, subtly different explanations can yield widely varying downstream task accuracy. Explanations that have not been “tuned” for a task, such as off-the-shelf explanations written by non-experts, may lead to mediocre performance. This paper tackles the problem of how to optimize explanation-infused prompts in a blackbox fashion. We first generate sets of candidate explanations for each example in the prompt using a leave-one-out scheme, then find an effective combination of these explanations with a two-stage framework. We first evaluate explanations for each in-context example in isolation according to two proxy metrics, log likelihood and accuracy on new examples. Then, we search over combinations of explanations to find one that yields high performance against a silver-labeled development set. Across four textual reasoning tasks spanning question answering, mathematical reasoning, and natural language inference, results show that our proxy metrics correlate with ground truth accuracy and our overall method can effectively improve prompts over crowdworker annotations and naive search strategies</abstract>
    <identifier type="citekey">ye-durrett-2023-explanation</identifier>
    <identifier type="doi">10.18653/v1/2023.emnlp-main.41</identifier>
    <location>
        <url>https://aclanthology.org/2023.emnlp-main.41</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>619</start>
            <end>637</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Explanation Selection Using Unlabeled Data for Chain-of-Thought Prompting
%A Ye, Xi
%A Durrett, Greg
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F ye-durrett-2023-explanation
%X Recent work has shown how to prompt large language models with explanations to obtain strong performance on textual reasoning tasks, i.e., the chain-of-thought paradigm. However, subtly different explanations can yield widely varying downstream task accuracy. Explanations that have not been “tuned” for a task, such as off-the-shelf explanations written by non-experts, may lead to mediocre performance. This paper tackles the problem of how to optimize explanation-infused prompts in a blackbox fashion. We first generate sets of candidate explanations for each example in the prompt using a leave-one-out scheme, then find an effective combination of these explanations with a two-stage framework. We first evaluate explanations for each in-context example in isolation according to two proxy metrics, log likelihood and accuracy on new examples. Then, we search over combinations of explanations to find one that yields high performance against a silver-labeled development set. Across four textual reasoning tasks spanning question answering, mathematical reasoning, and natural language inference, results show that our proxy metrics correlate with ground truth accuracy and our overall method can effectively improve prompts over crowdworker annotations and naive search strategies
%R 10.18653/v1/2023.emnlp-main.41
%U https://aclanthology.org/2023.emnlp-main.41
%U https://doi.org/10.18653/v1/2023.emnlp-main.41
%P 619-637

Markdown (Informal)

[Explanation Selection Using Unlabeled Data for Chain-of-Thought Prompting](https://aclanthology.org/2023.emnlp-main.41) (Ye & Durrett, EMNLP 2023)

• Explanation Selection Using Unlabeled Data for Chain-of-Thought Prompting (Ye & Durrett, EMNLP 2023)

ACL

• Xi Ye and Greg Durrett. 2023. Explanation Selection Using Unlabeled Data for Chain-of-Thought Prompting. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 619–637, Singapore. Association for Computational Linguistics.