Keyphrase Generation: Lessons from a Reproducibility Study

Edwin Thomas, Sowmya Vajjala


Abstract
Reproducibility studies are treated as means to verify the validity of a scientific method, but what else can we learn from such experiments? We addressed this question taking Keyphrase Generation (KPG) as the use case in this paper, by studying three state-of-the-art KPG models in terms of reproducibility under either the same (same data/model/code) or varied (different training data/model, but same code) conditions, and exploring different ways of comparing KPG models beyond the most commonly used evaluation measures. We drew some conclusions on the state of the art in KPG based on these experiments, and provided guidelines for researchers working on the topic about reporting experimental results in a more comprehensive manner.
Anthology ID:
2024.lrec-main.849
Volume:
Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
Month:
May
Year:
2024
Address:
Torino, Italia
Editors:
Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, Nianwen Xue
Venues:
LREC | COLING
SIG:
Publisher:
ELRA and ICCL
Note:
Pages:
9720–9731
Language:
URL:
https://aclanthology.org/2024.lrec-main.849
DOI:
Bibkey:
Cite (ACL):
Edwin Thomas and Sowmya Vajjala. 2024. Keyphrase Generation: Lessons from a Reproducibility Study. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 9720–9731, Torino, Italia. ELRA and ICCL.
Cite (Informal):
Keyphrase Generation: Lessons from a Reproducibility Study (Thomas & Vajjala, LREC-COLING 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.lrec-main.849.pdf