HI-CMLM: Improve CMLM with Hybrid Decoder Input

Minghan Wang, Guo Jiaxin, Yuxia Wang, Yimeng Chen, Su Chang, Daimeng Wei, Min Zhang, Shimin Tao, Hao Yang


Abstract
Mask-predict CMLM (Ghazvininejad et al.,2019) has achieved stunning performance among non-autoregressive NMT models, but we find that the mechanism of predicting all of the target words only depending on the hidden state of [MASK] is not effective and efficient in initial iterations of refinement, resulting in ungrammatical repetitions and slow convergence. In this work, we mitigate this problem by combining copied source with embeddings of [MASK] in decoder. Notably. it’s not a straightforward copying that is shown to be useless, but a novel heuristic hybrid strategy — fence-mask. Experimental results show that it gains consistent boosts on both WMT14 En<->De and WMT16 En<->Ro corpus by 0.5 BLEU on average, and 1 BLEU for less-informative short sentences. This reveals that incorporating additional information by proper strategies is beneficial to improve CMLM, particularly translation quality of short texts and speeding up early-stage convergence.
Anthology ID:
2021.inlg-1.16
Volume:
Proceedings of the 14th International Conference on Natural Language Generation
Month:
August
Year:
2021
Address:
Aberdeen, Scotland, UK
Editors:
Anya Belz, Angela Fan, Ehud Reiter, Yaji Sripada
Venue:
INLG
SIG:
SIGGEN
Publisher:
Association for Computational Linguistics
Note:
Pages:
167–171
Language:
URL:
https://aclanthology.org/2021.inlg-1.16
DOI:
10.18653/v1/2021.inlg-1.16
Bibkey:
Cite (ACL):
Minghan Wang, Guo Jiaxin, Yuxia Wang, Yimeng Chen, Su Chang, Daimeng Wei, Min Zhang, Shimin Tao, and Hao Yang. 2021. HI-CMLM: Improve CMLM with Hybrid Decoder Input. In Proceedings of the 14th International Conference on Natural Language Generation, pages 167–171, Aberdeen, Scotland, UK. Association for Computational Linguistics.
Cite (Informal):
HI-CMLM: Improve CMLM with Hybrid Decoder Input (Wang et al., INLG 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.inlg-1.16.pdf