CLMSM: A Multi-Task Learning Framework for Pre-training on Procedural Text

Abhilash Nandy, Manav Kapadnis, Pawan Goyal, Niloy Ganguly


Abstract
In this paper, we propose ***CLMSM***, a domain-specific, continual pre-training framework, that learns from a large set of procedural recipes. ***CLMSM*** uses a Multi-Task Learning Framework to optimize two objectives - a) Contrastive Learning using hard triplets to learn fine-grained differences across entities in the procedures, and b) a novel Mask-Step Modelling objective to learn step-wise context of a procedure. We test the performance of ***CLMSM*** on the downstream tasks of tracking entities and aligning actions between two procedures on three datasets, one of which is an open-domain dataset not conforming with the pre-training dataset. We show that ***CLMSM*** not only outperforms baselines on recipes (in-domain) but is also able to generalize to open-domain procedural NLP tasks.
Anthology ID:
2023.findings-emnlp.589
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2023
Month:
December
Year:
2023
Address:
Singapore
Editors:
Houda Bouamor, Juan Pino, Kalika Bali
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
8793–8806
Language:
URL:
https://aclanthology.org/2023.findings-emnlp.589
DOI:
10.18653/v1/2023.findings-emnlp.589
Bibkey:
Cite (ACL):
Abhilash Nandy, Manav Kapadnis, Pawan Goyal, and Niloy Ganguly. 2023. CLMSM: A Multi-Task Learning Framework for Pre-training on Procedural Text. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8793–8806, Singapore. Association for Computational Linguistics.
Cite (Informal):
CLMSM: A Multi-Task Learning Framework for Pre-training on Procedural Text (Nandy et al., Findings 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.findings-emnlp.589.pdf