<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="zhang-li-2023-large">
    <titleInfo>
        <title>Can Large Language Model Comprehend Ancient Chinese? A Preliminary Test on ACLUE</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Yixuan</namePart>
        <namePart type="family">Zhang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Haonan</namePart>
        <namePart type="family">Li</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-09</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Ancient Language Processing Workshop</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Adam</namePart>
            <namePart type="family">Anderson</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Shai</namePart>
            <namePart type="family">Gordin</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Bin</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yudong</namePart>
            <namePart type="family">Liu</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Marco</namePart>
            <namePart type="given">C</namePart>
            <namePart type="family">Passarotti</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>INCOMA Ltd., Shoumen, Bulgaria</publisher>
            <place>
                <placeTerm type="text">Varna, Bulgaria</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Large language models (LLMs) have demonstrated exceptional language understanding and generation capabilities. However, their ability to comprehend ancient languages, specifically ancient Chinese, remains largely unexplored. To bridge this gap, we introduce ACLUE, an evaluation benchmark designed to assess the language abilities of models in relation to ancient Chinese. ACLUE consists of 15 tasks that cover a range of skills, including phonetic, lexical, syntactic, semantic, inference and knowledge. By evaluating 8 state-of-the-art multilingual and Chinese LLMs, we have observed a significant divergence in their performance between modern Chinese and ancient Chinese. Among the evaluated models, ChatGLM2 demonstrates the highest level of performance, achieving an average accuracy of 37.45%. We have established a leaderboard for communities to assess their models.</abstract>
    <identifier type="citekey">zhang-li-2023-large</identifier>
    <location>
        <url>https://aclanthology.org/2023.alp-1.9/</url>
    </location>
    <part>
        <date>2023-09</date>
        <extent unit="page">
            <start>80</start>
            <end>87</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Can Large Language Model Comprehend Ancient Chinese? A Preliminary Test on ACLUE
%A Zhang, Yixuan
%A Li, Haonan
%Y Anderson, Adam
%Y Gordin, Shai
%Y Li, Bin
%Y Liu, Yudong
%Y Passarotti, Marco C.
%S Proceedings of the Ancient Language Processing Workshop
%D 2023
%8 September
%I INCOMA Ltd., Shoumen, Bulgaria
%C Varna, Bulgaria
%F zhang-li-2023-large
%X Large language models (LLMs) have demonstrated exceptional language understanding and generation capabilities. However, their ability to comprehend ancient languages, specifically ancient Chinese, remains largely unexplored. To bridge this gap, we introduce ACLUE, an evaluation benchmark designed to assess the language abilities of models in relation to ancient Chinese. ACLUE consists of 15 tasks that cover a range of skills, including phonetic, lexical, syntactic, semantic, inference and knowledge. By evaluating 8 state-of-the-art multilingual and Chinese LLMs, we have observed a significant divergence in their performance between modern Chinese and ancient Chinese. Among the evaluated models, ChatGLM2 demonstrates the highest level of performance, achieving an average accuracy of 37.45%. We have established a leaderboard for communities to assess their models.
%U https://aclanthology.org/2023.alp-1.9/
%P 80-87

Markdown (Informal)

[Can Large Language Model Comprehend Ancient Chinese? A Preliminary Test on ACLUE](https://aclanthology.org/2023.alp-1.9/) (Zhang & Li, ALP 2023)

• Can Large Language Model Comprehend Ancient Chinese? A Preliminary Test on ACLUE (Zhang & Li, ALP 2023)

ACL

• Yixuan Zhang and Haonan Li. 2023. Can Large Language Model Comprehend Ancient Chinese? A Preliminary Test on ACLUE. In Proceedings of the Ancient Language Processing Workshop, pages 80–87, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria.