<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="dou-peng-2021-improving">
    <titleInfo>
        <title>Improving Pre-trained Vision-and-Language Embeddings for Phrase Grounding</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Zi-Yi</namePart>
        <namePart type="family">Dou</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Nanyun</namePart>
        <namePart type="family">Peng</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Marie-Francine</namePart>
            <namePart type="family">Moens</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Xuanjing</namePart>
            <namePart type="family">Huang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Lucia</namePart>
            <namePart type="family">Specia</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Scott</namePart>
            <namePart type="given">Wen-tau</namePart>
            <namePart type="family">Yih</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online and Punta Cana, Dominican Republic</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Phrase grounding aims to map textual phrases to their associated image regions, which can be a prerequisite for multimodal reasoning and can benefit tasks requiring identifying objects based on language. With pre-trained vision-and-language models achieving impressive performance across tasks, it remains unclear if we can directly utilize their learned embeddings for phrase grounding without fine-tuning. To this end, we propose a method to extract matched phrase-region pairs from pre-trained vision-and-language embeddings and propose four fine-tuning objectives to improve the model phrase grounding ability using image-caption data without any supervised grounding signals. Experiments on two representative datasets demonstrate the effectiveness of our objectives, outperforming baseline models in both weakly-supervised and supervised phrase grounding settings. In addition, we evaluate the aligned embeddings on several other downstream tasks and show that we can achieve better phrase grounding without sacrificing representation generality.</abstract>
    <identifier type="citekey">dou-peng-2021-improving</identifier>
    <identifier type="doi">10.18653/v1/2021.emnlp-main.513</identifier>
    <location>
        <url>https://aclanthology.org/2021.emnlp-main.513/</url>
    </location>
    <part>
        <date>2021-11</date>
        <extent unit="page">
            <start>6362</start>
            <end>6371</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Improving Pre-trained Vision-and-Language Embeddings for Phrase Grounding
%A Dou, Zi-Yi
%A Peng, Nanyun
%Y Moens, Marie-Francine
%Y Huang, Xuanjing
%Y Specia, Lucia
%Y Yih, Scott Wen-tau
%S Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
%D 2021
%8 November
%I Association for Computational Linguistics
%C Online and Punta Cana, Dominican Republic
%F dou-peng-2021-improving
%X Phrase grounding aims to map textual phrases to their associated image regions, which can be a prerequisite for multimodal reasoning and can benefit tasks requiring identifying objects based on language. With pre-trained vision-and-language models achieving impressive performance across tasks, it remains unclear if we can directly utilize their learned embeddings for phrase grounding without fine-tuning. To this end, we propose a method to extract matched phrase-region pairs from pre-trained vision-and-language embeddings and propose four fine-tuning objectives to improve the model phrase grounding ability using image-caption data without any supervised grounding signals. Experiments on two representative datasets demonstrate the effectiveness of our objectives, outperforming baseline models in both weakly-supervised and supervised phrase grounding settings. In addition, we evaluate the aligned embeddings on several other downstream tasks and show that we can achieve better phrase grounding without sacrificing representation generality.
%R 10.18653/v1/2021.emnlp-main.513
%U https://aclanthology.org/2021.emnlp-main.513/
%U https://doi.org/10.18653/v1/2021.emnlp-main.513
%P 6362-6371

Markdown (Informal)

[Improving Pre-trained Vision-and-Language Embeddings for Phrase Grounding](https://aclanthology.org/2021.emnlp-main.513/) (Dou & Peng, EMNLP 2021)

• Improving Pre-trained Vision-and-Language Embeddings for Phrase Grounding (Dou & Peng, EMNLP 2021)

ACL

• Zi-Yi Dou and Nanyun Peng. 2021. Improving Pre-trained Vision-and-Language Embeddings for Phrase Grounding. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6362–6371, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.