Complex Verbs are Different: Exploring the Visual Modality in Multi-Modal Models to Predict Compositionality

Maximilian Köper, Sabine Schulte im Walde


Abstract
This paper compares a neural network DSM relying on textual co-occurrences with a multi-modal model integrating visual information. We focus on nominal vs. verbal compounds, and zoom into lexical, empirical and perceptual target properties to explore the contribution of the visual modality. Our experiments show that (i) visual features contribute differently for verbs than for nouns, and (ii) images complement textual information, if (a) the textual modality by itself is poor and appropriate image subsets are used, or (b) the textual modality by itself is rich and large (potentially noisy) images are added.
Anthology ID:
W17-1728
Volume:
Proceedings of the 13th Workshop on Multiword Expressions (MWE 2017)
Month:
April
Year:
2017
Address:
Valencia, Spain
Editors:
Stella Markantonatou, Carlos Ramisch, Agata Savary, Veronika Vincze
Venue:
MWE
SIG:
SIGLEX
Publisher:
Association for Computational Linguistics
Note:
Pages:
200–206
Language:
URL:
https://aclanthology.org/W17-1728
DOI:
10.18653/v1/W17-1728
Bibkey:
Cite (ACL):
Maximilian Köper and Sabine Schulte im Walde. 2017. Complex Verbs are Different: Exploring the Visual Modality in Multi-Modal Models to Predict Compositionality. In Proceedings of the 13th Workshop on Multiword Expressions (MWE 2017), pages 200–206, Valencia, Spain. Association for Computational Linguistics.
Cite (Informal):
Complex Verbs are Different: Exploring the Visual Modality in Multi-Modal Models to Predict Compositionality (Köper & Schulte im Walde, MWE 2017)
Copy Citation:
PDF:
https://aclanthology.org/W17-1728.pdf