<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="agrawal-carpuat-2024-text">
    <titleInfo>
        <title>Do Text Simplification Systems Preserve Meaning? A Human Evaluation via Reading Comprehension</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Sweta</namePart>
        <namePart type="family">Agrawal</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Marine</namePart>
        <namePart type="family">Carpuat</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Automatic text simplification (TS) aims to automate the process of rewriting text to make it easier for people to read. A pre-requisite for TS to be useful is that it should convey information that is consistent with the meaning of the original text. However, current TS evaluation protocols assess system outputs for simplicity and meaning preservation without regard for the document context in which output sentences occur and for how people understand them. In this work, we introduce a human evaluation framework to assess whether simplified texts preserve meaning using reading comprehension questions. With this framework, we conduct a thorough human evaluation of texts by humans and by nine automatic systems. Supervised systems that leverage pre-training knowledge achieve the highest scores on the reading comprehension tasks among the automatic controllable TS systems. However, even the best-performing supervised system struggles with at least 14% of the questions, marking them as “unanswerable” based on simplified content. We further investigate how existing TS evaluation metrics and automatic question-answering systems approximate the human judgments we obtained.</abstract>
    <identifier type="citekey">agrawal-carpuat-2024-text</identifier>
    <identifier type="doi">10.1162/tacl_a_00653</identifier>
    <location>
        <url>https://aclanthology.org/2024.tacl-1.24/</url>
    </location>
    <part>
        <date>2024</date>
        <detail type="volume"><number>12</number></detail>
        <extent unit="page">
            <start>432</start>
            <end>448</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T Do Text Simplification Systems Preserve Meaning? A Human Evaluation via Reading Comprehension
%A Agrawal, Sweta
%A Carpuat, Marine
%J Transactions of the Association for Computational Linguistics
%D 2024
%V 12
%I MIT Press
%C Cambridge, MA
%F agrawal-carpuat-2024-text
%X Automatic text simplification (TS) aims to automate the process of rewriting text to make it easier for people to read. A pre-requisite for TS to be useful is that it should convey information that is consistent with the meaning of the original text. However, current TS evaluation protocols assess system outputs for simplicity and meaning preservation without regard for the document context in which output sentences occur and for how people understand them. In this work, we introduce a human evaluation framework to assess whether simplified texts preserve meaning using reading comprehension questions. With this framework, we conduct a thorough human evaluation of texts by humans and by nine automatic systems. Supervised systems that leverage pre-training knowledge achieve the highest scores on the reading comprehension tasks among the automatic controllable TS systems. However, even the best-performing supervised system struggles with at least 14% of the questions, marking them as “unanswerable” based on simplified content. We further investigate how existing TS evaluation metrics and automatic question-answering systems approximate the human judgments we obtained.
%R 10.1162/tacl_a_00653
%U https://aclanthology.org/2024.tacl-1.24/
%U https://doi.org/10.1162/tacl_a_00653
%P 432-448

Markdown (Informal)

[Do Text Simplification Systems Preserve Meaning? A Human Evaluation via Reading Comprehension](https://aclanthology.org/2024.tacl-1.24/) (Agrawal & Carpuat, TACL 2024)

• Do Text Simplification Systems Preserve Meaning? A Human Evaluation via Reading Comprehension (Agrawal & Carpuat, TACL 2024)

ACL

• Sweta Agrawal and Marine Carpuat. 2024. Do Text Simplification Systems Preserve Meaning? A Human Evaluation via Reading Comprehension. Transactions of the Association for Computational Linguistics, 12:432–448.