MATTER: Memory-Augmented Transformer Using Heterogeneous Knowledge Sources

Dongkyu Lee, Chandana Satya Prakash, Jack FitzGerald, Jens Lehmann


Abstract
Leveraging external knowledge is crucial for achieving high performance in knowledge-intensive tasks, such as question answering. The retrieve-and-read approach is widely adopted for integrating external knowledge into a language model. However, this approach suffers from increased computational cost and latency due to the long context length, which grows proportionally with the number of retrieved knowledge. Furthermore, existing retrieval-augmented models typically retrieve information from a single type of knowledge source, limiting their scalability to diverse knowledge sources with varying structures. In this work, we introduce an efficient memory-augmented transformer called MATTER, designed to retrieve relevant knowledge from multiple heterogeneous knowledge sources. Specifically, our model retrieves and reads from both unstructured sources (paragraphs) and semi-structured sources (QA pairs) in the form of fixed-length neural memories. We demonstrate that our model outperforms existing efficient retrieval-augmented models on popular QA benchmarks in terms of both accuracy and speed. Furthermore, MATTER achieves competitive results compared to conventional read-and-retrieve models while having 100x throughput during inference.
Anthology ID:
2024.findings-acl.953
Volume:
Findings of the Association for Computational Linguistics: ACL 2024
Month:
August
Year:
2024
Address:
Bangkok, Thailand
Editors:
Lun-Wei Ku, Andre Martins, Vivek Srikumar
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
16110–16121
Language:
URL:
https://aclanthology.org/2024.findings-acl.953
DOI:
10.18653/v1/2024.findings-acl.953
Bibkey:
Cite (ACL):
Dongkyu Lee, Chandana Satya Prakash, Jack FitzGerald, and Jens Lehmann. 2024. MATTER: Memory-Augmented Transformer Using Heterogeneous Knowledge Sources. In Findings of the Association for Computational Linguistics: ACL 2024, pages 16110–16121, Bangkok, Thailand. Association for Computational Linguistics.
Cite (Informal):
MATTER: Memory-Augmented Transformer Using Heterogeneous Knowledge Sources (Lee et al., Findings 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.findings-acl.953.pdf