Transcription of Self-Attention with Relative Position Representations
{{id}} {{{paragraph}}}
Example: tourism industry
Self-Attention with Relative Position Representations
Self-Attention with Relative Position RepresentationsPeter UszkoreitGoogle VaswaniGoogle entirely on an attention mechanism,the Transformer introduced by Vaswani etal. (2017) achieves state-of-the-art results formachine translation. In contrast to recurrentand convolutional neural networks, it doesnot explicitly model Relative or absolute po-sition information in its ,it requires adding Representations of abso-lute positions to its this workwe present an alternative approach, extend-ing the Self-Attention mechanism to efficientlyconsider Representations of the Relative posi-tions, or distances between sequence the WMT 2014 English-to-German andEnglish-to-French translation tasks, this ap-proach yields improvements of BLEU BLEU over absolute Position representa-tions, respectively.
on distance (Parikh et al.,2016). In this work we present an efficient way of incorporating relative position representations in the self-attention mechanism of the Transformer. Even when entirely replacing its absolute position encodings, we demonstrate significant improve-ments in translation quality on two machine trans-lation tasks.
Loading..
Tags:
Information
Domain:
Source:
Link to this page:
Please notify us if you found a problem with this document: