Transcription of Self-Attention with Relative Position Representations
{{id}} {{{paragraph}}}
Example: marketing
Self-Attention with Relative Position Representations
Self-Attention with Relative Position RepresentationsPeter UszkoreitGoogle VaswaniGoogle entirely on an attention mechanism,the Transformer introduced by Vaswani etal. (2017) achieves state-of-the-art results formachine translation. In contrast to recurrentand convolutional neural networks, it doesnot explicitly model Relative or absolute po-sition information in its ,it requires adding Representations of abso-lute positions to its this workwe present an alternative approach, extend-ing the Self-Attention mechanism to efficientlyconsider Representations of the Relative posi-tions, or distances between sequence the WMT 2014 English-to-German andEnglish-to-French translation tasks, this ap-proach yields improvements of BLEU BLEU over absolute Position representa-tions, respectively. Notably, we observe thatcombining Relative and absolute Position rep-resentations yields no further improvement intranslation quality.
representations of position information is an espe-cially important consideration since the model is otherwise entirely invariant to sequence ordering. Attention-based models have therefore used posi-tion encodings or biased attention weights based on distance (Parikh et al.,2016).
Loading..
Tags:
Information
Domain:
Source:
Link to this page:
Please notify us if you found a problem with this document: