Investigating Content Planning for Navigating Trade-offs in Knowledge-Grounded Dialogue

Kushal Chawla, Hannah Rashkin, Gaurav Singh Tomar, David Reitter


Abstract
Knowledge-grounded dialogue generation is a challenging task because it requires satisfying two fundamental, yet often competing constraints: being responsive in a manner that is specific to what the conversation partner has said while also being attributable to an underlying source document. In this work, we bring this trade-off between these two objectives (specificity and attribution) to light, and ask the question: Can explicit content planning before the response generation help the model to address this challenge? To answer this question, we design a framework called PLEDGE, which allows us to experiment with various plan variables explored in prior work supporting both metric-agnostic and metric-aware approaches. While content planning shows promise, our results on whether it can actually help to navigate this trade-off are mixed – planning mechanisms that are metric-aware (use automatic metrics during training) are better at automatic evaluations but underperform in human judgment compared to metric-agnostic mechanisms. We discuss how this may be caused by over-fitting to automatic metrics, and the need for future work to better calibrate these metrics towards human judgment. We hope the observations from our analysis will inform future work that aims to apply content planning in this context.
Anthology ID:
2024.eacl-long.142
Volume:
Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
March
Year:
2024
Address:
St. Julian’s, Malta
Editors:
Yvette Graham, Matthew Purver
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2316–2335
Language:
URL:
https://aclanthology.org/2024.eacl-long.142
DOI:
Bibkey:
Cite (ACL):
Kushal Chawla, Hannah Rashkin, Gaurav Singh Tomar, and David Reitter. 2024. Investigating Content Planning for Navigating Trade-offs in Knowledge-Grounded Dialogue. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2316–2335, St. Julian’s, Malta. Association for Computational Linguistics.
Cite (Informal):
Investigating Content Planning for Navigating Trade-offs in Knowledge-Grounded Dialogue (Chawla et al., EACL 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.eacl-long.142.pdf
Video:
 https://aclanthology.org/2024.eacl-long.142.mp4