Linformer: Self-Attention with Linear Complexity
Linformer: Self-Attention with Linear ComplexitySinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, Hao MaFacebook AI, Seattle, WA{sinongwang, belindali, hanfang, mkhabsa, transformer models have shown extraordinary success in achieving state-of-the-art results in many natural language processing applications. However, trainingand deploying these models can be prohibitively costly for long sequences, asthe standard Self-Attention mechanism of the Transformer usesO(n2)time andspace with respect to sequence length. In this paper, we demonstrate that theself-attention mechanism can be approximated by a low-rank matrix. We furtherexploit this finding to propose a new Self-Attention mechanism, which reducesthe overall Self-Attention Complexity fromO(n2)toO(n)in both time and resulting Linear transformer, theLinformer, performs on par with standardTransformer models, while being much more memory- and IntroductionTransformer models (Vaswani et al.)}
complexity to O(nlog(n)). However, in practice, their complexity term has a large constant 1282 and it is only more efficient than the vanilla transformer when sequence length is extremely long. Improving Optimizer Efficiency: Microbatching (Huang et al., 2019) splits a batch into small
Download Linformer: Self-Attention with Linear Complexity
Information
Domain:
Source:
Link to this page:
Please notify us if you found a problem with this document: