C-MORE: Pretraining to Answer Open-Domain Questions by Consulting Millions of References

Xiang Yue, Xiaoman Pan, Wenlin Yao, Dian Yu, Dong Yu, Jianshu Chen


Abstract
We consider the problem of pretraining a two-stage open-domain question answering (QA) system (retriever + reader) with strong transfer capabilities. The key challenge is how to construct a large amount of high-quality question-answer-context triplets without task-specific annotations. Specifically, the triplets should align well with downstream tasks by: (i) covering a wide range of domains (for open-domain applications), (ii) linking a question to its semantically relevant context with supporting evidence (for training the retriever), and (iii) identifying the correct answer in the context (for training the reader). Previous pretraining approaches generally fall short of one or more of these requirements. In this work, we automatically construct a large-scale corpus that meets all three criteria by consulting millions of references cited within Wikipedia. The well-aligned pretraining signals benefit both the retriever and the reader significantly. Our pretrained retriever leads to 2%-10% absolute gains in top-20 accuracy. And with our pretrained reader, the entire system improves by up to 4% in exact match.
Anthology ID:
2022.acl-short.41
Volume:
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
371–377
Language:
URL:
https://aclanthology.org/2022.acl-short.41
DOI:
10.18653/v1/2022.acl-short.41
Bibkey:
Cite (ACL):
Xiang Yue, Xiaoman Pan, Wenlin Yao, Dian Yu, Dong Yu, and Jianshu Chen. 2022. C-MORE: Pretraining to Answer Open-Domain Questions by Consulting Millions of References. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 371–377, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
C-MORE: Pretraining to Answer Open-Domain Questions by Consulting Millions of References (Yue et al., ACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.acl-short.41.pdf
Video:
 https://aclanthology.org/2022.acl-short.41.mp4
Code
 xiangyue9607/c-more
Data
Natural QuestionsTriviaQA