Passage-Mask: A Learnable Regularization Strategy for Retriever-Reader Models

Shujian Zhang, Chengyue Gong, Xingchao Liu


Abstract
Retriever-reader models achieve competitive performance across many different NLP tasks such as open question answering and dialogue conversations. In this work, we notice these models easily overfit the top-rank retrieval passages and standard training fails to reason over the entire retrieval passages. We introduce a learnable passage mask mechanism which desensitizes the impact from the top-rank retrieval passages and prevents the model from overfitting. Controlling the gradient variance with fewer mask candidates and selecting the mask candidates with one-shot bi-level optimization, our learnable regularization strategy enforces the answer generation to focus on the entire retrieval passages. Experiments on different tasks across open question answering, dialogue conversation, and fact verification show that our method consistently outperforms its baselines. Extensive experiments and ablation studies demonstrate that our method can be general, effective, and beneficial for many NLP tasks.
Anthology ID:
2022.emnlp-main.260
Volume:
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates
Editors:
Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3931–3943
Language:
URL:
https://aclanthology.org/2022.emnlp-main.260
DOI:
10.18653/v1/2022.emnlp-main.260
Bibkey:
Cite (ACL):
Shujian Zhang, Chengyue Gong, and Xingchao Liu. 2022. Passage-Mask: A Learnable Regularization Strategy for Retriever-Reader Models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3931–3943, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cite (Informal):
Passage-Mask: A Learnable Regularization Strategy for Retriever-Reader Models (Zhang et al., EMNLP 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.emnlp-main.260.pdf