Nikolai Gerasimenko
2024
Your Transformer is Secretly Linear
Anton Razzhigaev
|
Matvey Mikhalchuk
|
Elizaveta Goncharova
|
Nikolai Gerasimenko
|
Ivan Oseledets
|
Denis Dimitrov
|
Andrey Kuznetsov
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
This paper reveals a novel linear characteristic exclusive to transformer decoders, including models like GPT, LLaMA, OPT, BLOOM and others. We analyze embedding transformations between sequential layers, uncovering an almost perfect linear relationship (Procrustes similarity score of 0.99). However, linearity decreases when the residual component is removed, due to a consistently low transformer layer output norm. Our experiments show that pruning or linearly approximating some of the layers does not impact loss or model performance significantly. Moreover, we introduce a cosine-similarity-based regularization in our pretraining experiments on smaller models, aimed at reducing layer linearity. This regularization not only improves performance metrics on benchmarks like Tiny Stories and SuperGLUE but as well successfully decreases the linearity of the models. This study challenges the existing understanding of transformer architectures, suggesting that their operation may be more linear than previously assumed.
Search
Co-authors
- Anton Razzhigaev 1
- Matvey Mikhalchuk 1
- Elizaveta Goncharova 1
- Ivan Oseledets 1
- Denis Dimitrov 1
- show all...
Venues
- acl1