Analysing Word Representation from the Input and Output Embeddings in Neural Network Language Models

Steven Derby, Paul Miller, Barry Devereux


Abstract
Researchers have recently demonstrated that tying the neural weights between the input look-up table and the output classification layer can improve training and lower perplexity on sequence learning tasks such as language modelling. Such a procedure is possible due to the design of the softmax classification layer, which previous work has shown to comprise a viable set of semantic representations for the model vocabulary, and these these output embeddings are known to perform well on word similarity benchmarks. In this paper, we make meaningful comparisons between the input and output embeddings and other SOTA distributional models to gain a better understanding of the types of information they represent. We also construct a new set of word embeddings using the output embeddings to create locally-optimal approximations for the intermediate representations from the language model. These locally-optimal embeddings demonstrate excellent performance across all our evaluations.
Anthology ID:
2020.conll-1.36
Volume:
Proceedings of the 24th Conference on Computational Natural Language Learning
Month:
November
Year:
2020
Address:
Online
Editors:
Raquel Fernández, Tal Linzen
Venue:
CoNLL
SIG:
SIGNLL
Publisher:
Association for Computational Linguistics
Note:
Pages:
442–454
Language:
URL:
https://aclanthology.org/2020.conll-1.36
DOI:
10.18653/v1/2020.conll-1.36
Bibkey:
Cite (ACL):
Steven Derby, Paul Miller, and Barry Devereux. 2020. Analysing Word Representation from the Input and Output Embeddings in Neural Network Language Models. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 442–454, Online. Association for Computational Linguistics.
Cite (Informal):
Analysing Word Representation from the Input and Output Embeddings in Neural Network Language Models (Derby et al., CoNLL 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.conll-1.36.pdf
Optional supplementary material:
 2020.conll-1.36.OptionalSupplementaryMaterial.pdf
Code
 stevend94/conll2020
Data
MPQA Opinion CorpusPenn TreebankSICKSSTSST-2SST-5