<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="cai-etal-2023-masked">
    <titleInfo>
        <title>Masked Audio Text Encoders are Effective Multi-Modal Rescorers</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Jinglun</namePart>
        <namePart type="family">Cai</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Monica</namePart>
        <namePart type="family">Sunkara</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Xilai</namePart>
        <namePart type="family">Li</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Anshu</namePart>
        <namePart type="family">Bhatia</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Xiao</namePart>
        <namePart type="family">Pan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Sravan</namePart>
        <namePart type="family">Bodapati</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: ACL 2023</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Anna</namePart>
            <namePart type="family">Rogers</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jordan</namePart>
            <namePart type="family">Boyd-Graber</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Naoaki</namePart>
            <namePart type="family">Okazaki</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Toronto, Canada</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Masked Language Models (MLMs) have proven to be effective for second-pass rescoring in Automatic Speech Recognition (ASR) systems. In this work, we propose Masked Audio Text Encoder (MATE), a multi-modal masked language model rescorer which incorporates acoustic representations into the input space of MLM. We adopt contrastive learning for effectively aligning the modalities by learning shared representations. We show that using a multi-modal rescorer is beneficial for domain generalization of the ASR system when target domain data is unavailable. MATE reduces word error rate (WER) by 4%-16% on in-domain, and 3%-7% on out-of-domain datasets, over the text-only baseline. Additionally, with very limited amount of training data (0.8 hours) MATE achieves a WER reduction of 8%-23% over the first-pass baseline.</abstract>
    <identifier type="citekey">cai-etal-2023-masked</identifier>
    <identifier type="doi">10.18653/v1/2023.findings-acl.682</identifier>
    <location>
        <url>https://aclanthology.org/2023.findings-acl.682</url>
    </location>
    <part>
        <date>2023-07</date>
        <extent unit="page">
            <start>10718</start>
            <end>10730</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Masked Audio Text Encoders are Effective Multi-Modal Rescorers
%A Cai, Jinglun
%A Sunkara, Monica
%A Li, Xilai
%A Bhatia, Anshu
%A Pan, Xiao
%A Bodapati, Sravan
%Y Rogers, Anna
%Y Boyd-Graber, Jordan
%Y Okazaki, Naoaki
%S Findings of the Association for Computational Linguistics: ACL 2023
%D 2023
%8 July
%I Association for Computational Linguistics
%C Toronto, Canada
%F cai-etal-2023-masked
%X Masked Language Models (MLMs) have proven to be effective for second-pass rescoring in Automatic Speech Recognition (ASR) systems. In this work, we propose Masked Audio Text Encoder (MATE), a multi-modal masked language model rescorer which incorporates acoustic representations into the input space of MLM. We adopt contrastive learning for effectively aligning the modalities by learning shared representations. We show that using a multi-modal rescorer is beneficial for domain generalization of the ASR system when target domain data is unavailable. MATE reduces word error rate (WER) by 4%-16% on in-domain, and 3%-7% on out-of-domain datasets, over the text-only baseline. Additionally, with very limited amount of training data (0.8 hours) MATE achieves a WER reduction of 8%-23% over the first-pass baseline.
%R 10.18653/v1/2023.findings-acl.682
%U https://aclanthology.org/2023.findings-acl.682
%U https://doi.org/10.18653/v1/2023.findings-acl.682
%P 10718-10730

Markdown (Informal)

[Masked Audio Text Encoders are Effective Multi-Modal Rescorers](https://aclanthology.org/2023.findings-acl.682) (Cai et al., Findings 2023)

• Masked Audio Text Encoders are Effective Multi-Modal Rescorers (Cai et al., Findings 2023)

ACL

• Jinglun Cai, Monica Sunkara, Xilai Li, Anshu Bhatia, Xiao Pan, and Sravan Bodapati. 2023. Masked Audio Text Encoders are Effective Multi-Modal Rescorers. In Findings of the Association for Computational Linguistics: ACL 2023, pages 10718–10730, Toronto, Canada. Association for Computational Linguistics.