<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="raganato-etal-2020-fixed">
    <titleInfo>
        <title>Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Alessandro</namePart>
        <namePart type="family">Raganato</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yves</namePart>
        <namePart type="family">Scherrer</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jörg</namePart>
        <namePart type="family">Tiedemann</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2020</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Trevor</namePart>
            <namePart type="family">Cohn</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yulan</namePart>
            <namePart type="family">He</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yang</namePart>
            <namePart type="family">Liu</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Transformer-based models have brought a radical change to neural machine translation. A key feature of the Transformer architecture is the so-called multi-head attention mechanism, which allows the model to focus simultaneously on different parts of the input. However, recent works have shown that most attention heads learn simple, and often redundant, positional patterns. In this paper, we propose to replace all but one attention head of each encoder layer with simple fixed – non-learnable – attentive patterns that are solely based on position and do not require any external knowledge. Our experiments with different data sizes and multiple language pairs show that fixing the attention heads on the encoder side of the Transformer at training time does not impact the translation quality and even increases BLEU scores by up to 3 points in low-resource scenarios.</abstract>
    <identifier type="citekey">raganato-etal-2020-fixed</identifier>
    <identifier type="doi">10.18653/v1/2020.findings-emnlp.49</identifier>
    <location>
        <url>https://aclanthology.org/2020.findings-emnlp.49</url>
    </location>
    <part>
        <date>2020-11</date>
        <extent unit="page">
            <start>556</start>
            <end>568</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation
%A Raganato, Alessandro
%A Scherrer, Yves
%A Tiedemann, Jörg
%Y Cohn, Trevor
%Y He, Yulan
%Y Liu, Yang
%S Findings of the Association for Computational Linguistics: EMNLP 2020
%D 2020
%8 November
%I Association for Computational Linguistics
%C Online
%F raganato-etal-2020-fixed
%X Transformer-based models have brought a radical change to neural machine translation. A key feature of the Transformer architecture is the so-called multi-head attention mechanism, which allows the model to focus simultaneously on different parts of the input. However, recent works have shown that most attention heads learn simple, and often redundant, positional patterns. In this paper, we propose to replace all but one attention head of each encoder layer with simple fixed – non-learnable – attentive patterns that are solely based on position and do not require any external knowledge. Our experiments with different data sizes and multiple language pairs show that fixing the attention heads on the encoder side of the Transformer at training time does not impact the translation quality and even increases BLEU scores by up to 3 points in low-resource scenarios.
%R 10.18653/v1/2020.findings-emnlp.49
%U https://aclanthology.org/2020.findings-emnlp.49
%U https://doi.org/10.18653/v1/2020.findings-emnlp.49
%P 556-568

Markdown (Informal)

[Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation](https://aclanthology.org/2020.findings-emnlp.49) (Raganato et al., Findings 2020)

• Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation (Raganato et al., Findings 2020)

ACL

• Alessandro Raganato, Yves Scherrer, and Jörg Tiedemann. 2020. Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 556–568, Online. Association for Computational Linguistics.