Error-Sensitive Evaluation for Ordinal Target Variables

David Chen, Maury Courtland, Adam Faulkner, Aysu Ezen-Can


Abstract
Product reviews and satisfaction surveys seek customer feedback in the form of ranked scales. In these settings, widely used evaluation metrics including F1 and accuracy ignore the rank in the responses (e.g., ‘very likely’ is closer to ‘likely’ than ‘not at all’). In this paper, we hypothesize that the order of class values is important for evaluating classifiers on ordinal target variables and should not be disregarded. To test this hypothesis, we compared Multi-class Classification (MC) and Ordinal Regression (OR) by applying OR and MC to benchmark tasks involving ordinal target variables using the same underlying model architecture. Experimental results show that while MC outperformed OR for some datasets in accuracy and F1, OR is significantly better than MC for minimizing the error between prediction and target for all benchmarks, as revealed by error-sensitive metrics, e.g. mean-squared error (MSE) and Spearman correlation. Our findings motivate the need to establish consistent, error-sensitive metrics for evaluating benchmarks with ordinal target variables, and we hope that it stimulates interest in exploring alternative losses for ordinal problems.
Anthology ID:
2021.eval4nlp-1.19
Volume:
Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems
Month:
November
Year:
2021
Address:
Punta Cana, Dominican Republic
Editors:
Yang Gao, Steffen Eger, Wei Zhao, Piyawat Lertvittayakumjorn, Marina Fomicheva
Venue:
Eval4NLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
189–199
Language:
URL:
https://aclanthology.org/2021.eval4nlp-1.19
DOI:
10.18653/v1/2021.eval4nlp-1.19
Bibkey:
Cite (ACL):
David Chen, Maury Courtland, Adam Faulkner, and Aysu Ezen-Can. 2021. Error-Sensitive Evaluation for Ordinal Target Variables. In Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems, pages 189–199, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Error-Sensitive Evaluation for Ordinal Target Variables (Chen et al., Eval4NLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.eval4nlp-1.19.pdf
Video:
 https://aclanthology.org/2021.eval4nlp-1.19.mp4
Data
IMDb Movie ReviewsSSTSST-5