Question Difficulty – How to Estimate Without Norming, How to Use for Automated Grading

Ulrike Padó


Abstract
Question difficulty estimates guide test creation, but are too costly for small-scale testing. We empirically verify that Bloom’s Taxonomy, a standard tool for difficulty estimation during question creation, reliably predicts question difficulty observed after testing in a short-answer corpus. We also find that difficulty is mirrored in the amount of variation in student answers, which can be computed before grading. We show that question difficulty and its approximations are useful for automated grading, allowing us to identify the optimal feature set for grading each question even in an unseen-question setting.
Anthology ID:
W17-5001
Volume:
Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications
Month:
September
Year:
2017
Address:
Copenhagen, Denmark
Editors:
Joel Tetreault, Jill Burstein, Claudia Leacock, Helen Yannakoudakis
Venue:
BEA
SIG:
SIGEDU
Publisher:
Association for Computational Linguistics
Note:
Pages:
1–10
Language:
URL:
https://aclanthology.org/W17-5001
DOI:
10.18653/v1/W17-5001
Bibkey:
Cite (ACL):
Ulrike Padó. 2017. Question Difficulty – How to Estimate Without Norming, How to Use for Automated Grading. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pages 1–10, Copenhagen, Denmark. Association for Computational Linguistics.
Cite (Informal):
Question Difficulty – How to Estimate Without Norming, How to Use for Automated Grading (Padó, BEA 2017)
Copy Citation:
PDF:
https://aclanthology.org/W17-5001.pdf