<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="ceron-etal-2024-beyond">
    <titleInfo>
        <title>Beyond Prompt Brittleness: Evaluating the Reliability and Consistency of Political Worldviews in LLMs</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Tanise</namePart>
        <namePart type="family">Ceron</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Neele</namePart>
        <namePart type="family">Falk</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ana</namePart>
        <namePart type="family">Barić</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Dmitry</namePart>
        <namePart type="family">Nikolaev</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Sebastian</namePart>
        <namePart type="family">Padó</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Due to the widespread use of large language models (LLMs), we need to understand whether they embed a specific “worldview” and what these views reflect. Recent studies report that, prompted with political questionnaires, LLMs show left-liberal leanings (Feng et al., 2023; Motoki et al., 2024). However, it is as yet unclear whether these leanings are reliable (robust to prompt variations) and whether the leaning is consistent across policies and political leaning. We propose a series of tests which assess the reliability and consistency of LLMs’ stances on political statements based on a dataset of voting-advice questionnaires collected from seven EU countries and annotated for policy issues. We study LLMs ranging in size from 7B to 70B parameters and find that their reliability increases with parameter count. Larger models show overall stronger alignment with left-leaning parties but differ among policy programs: They show a (left-wing) positive stance towards environment protection, social welfare state, and liberal society but also (right-wing) law and order, with no consistent preferences in the areas of foreign policy and migration.</abstract>
    <identifier type="citekey">ceron-etal-2024-beyond</identifier>
    <identifier type="doi">10.1162/tacl_a_00710</identifier>
    <location>
        <url>https://aclanthology.org/2024.tacl-1.76/</url>
    </location>
    <part>
        <date>2024</date>
        <detail type="volume"><number>12</number></detail>
        <extent unit="page">
            <start>1378</start>
            <end>1400</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T Beyond Prompt Brittleness: Evaluating the Reliability and Consistency of Political Worldviews in LLMs
%A Ceron, Tanise
%A Falk, Neele
%A Barić, Ana
%A Nikolaev, Dmitry
%A Padó, Sebastian
%J Transactions of the Association for Computational Linguistics
%D 2024
%V 12
%I MIT Press
%C Cambridge, MA
%F ceron-etal-2024-beyond
%X Due to the widespread use of large language models (LLMs), we need to understand whether they embed a specific “worldview” and what these views reflect. Recent studies report that, prompted with political questionnaires, LLMs show left-liberal leanings (Feng et al., 2023; Motoki et al., 2024). However, it is as yet unclear whether these leanings are reliable (robust to prompt variations) and whether the leaning is consistent across policies and political leaning. We propose a series of tests which assess the reliability and consistency of LLMs’ stances on political statements based on a dataset of voting-advice questionnaires collected from seven EU countries and annotated for policy issues. We study LLMs ranging in size from 7B to 70B parameters and find that their reliability increases with parameter count. Larger models show overall stronger alignment with left-leaning parties but differ among policy programs: They show a (left-wing) positive stance towards environment protection, social welfare state, and liberal society but also (right-wing) law and order, with no consistent preferences in the areas of foreign policy and migration.
%R 10.1162/tacl_a_00710
%U https://aclanthology.org/2024.tacl-1.76/
%U https://doi.org/10.1162/tacl_a_00710
%P 1378-1400

Markdown (Informal)

[Beyond Prompt Brittleness: Evaluating the Reliability and Consistency of Political Worldviews in LLMs](https://aclanthology.org/2024.tacl-1.76/) (Ceron et al., TACL 2024)

• Beyond Prompt Brittleness: Evaluating the Reliability and Consistency of Political Worldviews in LLMs (Ceron et al., TACL 2024)

ACL

• Tanise Ceron, Neele Falk, Ana Barić, Dmitry Nikolaev, and Sebastian Padó. 2024. Beyond Prompt Brittleness: Evaluating the Reliability and Consistency of Political Worldviews in LLMs. Transactions of the Association for Computational Linguistics, 12:1378–1400.