CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modeling

Yu Bai, Xiyuan Zou, Heyan Huang, Sanxing Chen, Marc-Antoine Rondeau, Yang Gao, Jackie CK Cheung


Abstract
Long sequence modeling has gained broad interest as large language models (LLMs) continue to advance. Recent research has identified that a large portion of hidden states within the key-value caches of Transformer models can be discarded (also termed evicted) withoutaffecting the perplexity performance in generating long sequences. However, we show that these methods, despite preserving perplexity performance, often drop information that is important for solving downstream tasks, a problem which we call information neglect. To address this issue, we introduce Chunked Instruction-aware State Eviction (CItruS), a novel modeling technique that integrates the attention preferences useful for a downstream task into the eviction process of hidden states. In addition, we design a method for chunked sequence processing to further improve efficiency. Our training-free method exhibits superior performance on long sequence comprehension and retrieval tasks over several strong baselines under the same memory budget, while preserving language modeling perplexity. The code and data have been released at https://github.com/ybai-nlp/CItruS.
Anthology ID:
2024.emnlp-main.338
Volume:
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2024
Address:
Miami, Florida, USA
Editors:
Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
5908–5930
Language:
URL:
https://aclanthology.org/2024.emnlp-main.338
DOI:
10.18653/v1/2024.emnlp-main.338
Bibkey:
Cite (ACL):
Yu Bai, Xiyuan Zou, Heyan Huang, Sanxing Chen, Marc-Antoine Rondeau, Yang Gao, and Jackie CK Cheung. 2024. CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modeling. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5908–5930, Miami, Florida, USA. Association for Computational Linguistics.
Cite (Informal):
CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modeling (Bai et al., EMNLP 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.emnlp-main.338.pdf
Software:
 2024.emnlp-main.338.software.zip
Data:
 2024.emnlp-main.338.data.zip