Exploring the Use of Natural Language Descriptions of Intents for Large Language Models in Zero-shot Intent Classification

Taesuk Hong, Youbin Ahn, Dongkyu Lee, Joongbo Shin, Seungpil Won, Janghoon Han, Stanley Jungkyu Choi, Jungyun Seo


Abstract
In task-oriented dialogue systems, intent classification is crucial for accurately understanding user queries and providing appropriate services. This study explores the use of intent descriptions with large language models for unseen domain intent classification. By examining the effects of description quality, quantity, and input length management, we identify practical guidelines for optimizing performance. Our experiments using FLAN-T5 3B demonstrate that 1) high-quality descriptions for both training and testing significantly improve accuracy, 2) diversity in training descriptions doesn’t greatly affect performance, and 3) off-the-shelf rankers selecting around ten intent options reduce input length without compromising performance. We emphasize that high-quality testing descriptions have a greater impact on accuracy than training descriptions. These findings provide practical guidelines for using intent descriptions with large language models to achieve effective and efficient intent classification in low-resource settings.
Anthology ID:
2024.sigdial-1.39
Volume:
Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue
Month:
September
Year:
2024
Address:
Kyoto, Japan
Editors:
Tatsuya Kawahara, Vera Demberg, Stefan Ultes, Koji Inoue, Shikib Mehri, David Howcroft, Kazunori Komatani
Venue:
SIGDIAL
SIG:
SIGDIAL
Publisher:
Association for Computational Linguistics
Note:
Pages:
458–465
Language:
URL:
https://aclanthology.org/2024.sigdial-1.39
DOI:
10.18653/v1/2024.sigdial-1.39
Bibkey:
Cite (ACL):
Taesuk Hong, Youbin Ahn, Dongkyu Lee, Joongbo Shin, Seungpil Won, Janghoon Han, Stanley Jungkyu Choi, and Jungyun Seo. 2024. Exploring the Use of Natural Language Descriptions of Intents for Large Language Models in Zero-shot Intent Classification. In Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 458–465, Kyoto, Japan. Association for Computational Linguistics.
Cite (Informal):
Exploring the Use of Natural Language Descriptions of Intents for Large Language Models in Zero-shot Intent Classification (Hong et al., SIGDIAL 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.sigdial-1.39.pdf