Noot Noot at SemEval-2024 Task 7: Numerical Reasoning and Headline Generation

Sankalp Bahad, Yash Bhaskar, Parameswari Krishnamurthy


Abstract
Natural language processing (NLP) modelshave achieved remarkable progress in recentyears, particularly in tasks related to semanticanalysis. However, many existing benchmarksprimarily focus on lexical and syntactic un-derstanding, often overlooking the importanceof numerical reasoning abilities. In this pa-per, we argue for the necessity of incorporatingnumeral-awareness into NLP evaluations andpropose two distinct tasks to assess this capabil-ity: Numerical Reasoning and Headline Gener-ation. We present datasets curated for each taskand evaluate various approaches using both au-tomatic and human evaluation metrics. Ourresults demonstrate the diverse strategies em-ployed by participating teams and highlight thepromising performance of emerging modelslike Mixtral 8x7b instruct. We discuss the im-plications of our findings and suggest avenuesfor future research in advancing numeral-awarelanguage understanding and generation.
Anthology ID:
2024.semeval-1.131
Volume:
Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Atul Kr. Ojha, A. Seza Doğruöz, Harish Tayyar Madabushi, Giovanni Da San Martino, Sara Rosenthal, Aiala Rosá
Venue:
SemEval
SIG:
SIGLEX
Publisher:
Association for Computational Linguistics
Note:
Pages:
913–917
Language:
URL:
https://aclanthology.org/2024.semeval-1.131
DOI:
10.18653/v1/2024.semeval-1.131
Bibkey:
Cite (ACL):
Sankalp Bahad, Yash Bhaskar, and Parameswari Krishnamurthy. 2024. Noot Noot at SemEval-2024 Task 7: Numerical Reasoning and Headline Generation. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 913–917, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
Noot Noot at SemEval-2024 Task 7: Numerical Reasoning and Headline Generation (Bahad et al., SemEval 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.semeval-1.131.pdf
Supplementary material:
 2024.semeval-1.131.SupplementaryMaterial.txt