<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="srikanth-etal-2024-often">
    <titleInfo>
        <title>How Often Are Errors in Natural Language Reasoning Due to Paraphrastic Variability?</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Neha</namePart>
        <namePart type="family">Srikanth</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Marine</namePart>
        <namePart type="family">Carpuat</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Rachel</namePart>
        <namePart type="family">Rudinger</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Large language models have been shown to behave inconsistently in response to meaning-preserving paraphrastic inputs. At the same time, researchers evaluate the knowledge and reasoning abilities of these models with test evaluations that do not disaggregate the effect of paraphrastic variability on performance. We propose a metric, PC, for evaluating the paraphrastic consistency of natural language reasoning models based on the probability of a model achieving the same correctness on two paraphrases of the same problem. We mathematically connect this metric to the proportion of a model‘s variance in correctness attributable to paraphrasing. To estimate PC, we collect ParaNlu, a dataset of 7,782 human-written and validated paraphrased reasoning problems constructed on top of existing benchmark datasets for defeasible and abductive natural language inference.1 Using ParaNlu, we measure the paraphrastic consistency of several model classes and show that consistency dramatically increases with pretraining but not fine-tuning. All models tested exhibited room for improvement in paraphrastic consistency.</abstract>
    <identifier type="citekey">srikanth-etal-2024-often</identifier>
    <identifier type="doi">10.1162/tacl_a_00692</identifier>
    <location>
        <url>https://aclanthology.org/2024.tacl-1.63/</url>
    </location>
    <part>
        <date>2024</date>
        <detail type="volume"><number>12</number></detail>
        <extent unit="page">
            <start>1143</start>
            <end>1162</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T How Often Are Errors in Natural Language Reasoning Due to Paraphrastic Variability?
%A Srikanth, Neha
%A Carpuat, Marine
%A Rudinger, Rachel
%J Transactions of the Association for Computational Linguistics
%D 2024
%V 12
%I MIT Press
%C Cambridge, MA
%F srikanth-etal-2024-often
%X Large language models have been shown to behave inconsistently in response to meaning-preserving paraphrastic inputs. At the same time, researchers evaluate the knowledge and reasoning abilities of these models with test evaluations that do not disaggregate the effect of paraphrastic variability on performance. We propose a metric, PC, for evaluating the paraphrastic consistency of natural language reasoning models based on the probability of a model achieving the same correctness on two paraphrases of the same problem. We mathematically connect this metric to the proportion of a model‘s variance in correctness attributable to paraphrasing. To estimate PC, we collect ParaNlu, a dataset of 7,782 human-written and validated paraphrased reasoning problems constructed on top of existing benchmark datasets for defeasible and abductive natural language inference.1 Using ParaNlu, we measure the paraphrastic consistency of several model classes and show that consistency dramatically increases with pretraining but not fine-tuning. All models tested exhibited room for improvement in paraphrastic consistency.
%R 10.1162/tacl_a_00692
%U https://aclanthology.org/2024.tacl-1.63/
%U https://doi.org/10.1162/tacl_a_00692
%P 1143-1162

Markdown (Informal)

[How Often Are Errors in Natural Language Reasoning Due to Paraphrastic Variability?](https://aclanthology.org/2024.tacl-1.63/) (Srikanth et al., TACL 2024)

• How Often Are Errors in Natural Language Reasoning Due to Paraphrastic Variability? (Srikanth et al., TACL 2024)

ACL

• Neha Srikanth, Marine Carpuat, and Rachel Rudinger. 2024. How Often Are Errors in Natural Language Reasoning Due to Paraphrastic Variability?. Transactions of the Association for Computational Linguistics, 12:1143–1162.