factgenie: A Framework for Span-based Evaluation of Generated Texts

Zdeněk Kasner, Ondrej Platek, Patricia Schmidtova, Simone Balloccu, Ondrej Dusek


Abstract
We present ‘factgenie‘: a framework for annotating and visualizing word spans in textual model outputs. Annotations can capture various span-based phenomena such as semantic inaccuracies or irrelevant text. With ‘factgenie‘, the annotations can be collected both from human crowdworkers and large language models. Our framework consists of a web interface for data visualization and gathering text annotations, powered by an easily extensible codebase.
Anthology ID:
2024.inlg-demos.5
Volume:
Proceedings of the 17th International Natural Language Generation Conference: System Demonstrations
Month:
September
Year:
2024
Address:
Tokyo, Japan
Editors:
Saad Mahamood, Nguyen Le Minh, Daphne Ippolito
Venue:
INLG
SIG:
SIGGEN
Publisher:
Association for Computational Linguistics
Note:
Pages:
13–15
Language:
URL:
https://aclanthology.org/2024.inlg-demos.5
DOI:
Bibkey:
Cite (ACL):
Zdeněk Kasner, Ondrej Platek, Patricia Schmidtova, Simone Balloccu, and Ondrej Dusek. 2024. factgenie: A Framework for Span-based Evaluation of Generated Texts. In Proceedings of the 17th International Natural Language Generation Conference: System Demonstrations, pages 13–15, Tokyo, Japan. Association for Computational Linguistics.
Cite (Informal):
factgenie: A Framework for Span-based Evaluation of Generated Texts (Kasner et al., INLG 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.inlg-demos.5.pdf