Transcription of Self-Attention with Relative Position Representations
{{id}} {{{paragraph}}}
Example: biology
Self-Attention with Relative Position Representations
Self-Attention with Relative Position RepresentationsPeter UszkoreitGoogle VaswaniGoogle entirely on an attention mechanism,the Transformer introduced by Vaswani etal. (2017) achieves state-of-the-art results formachine translation. In contrast to recurrentand convolutional neural networks, it doesnot explicitly model Relative or absolute po-sition information in its ,it requires adding Representations of abso-lute positions to its this workwe present an alternative approach, extend-ing the Self-Attention mechanism to efficientlyconsider Representations of the Relative posi-tions, or distances between sequence the WMT 2014 English-to-German andEnglish-to-French translation tasks, this ap-proach yields improvements of BLEU BLEU over absolute Position representa-tions, respectively. Notably, we observe thatcombining Relative and absolute Position rep-resentations yields no further improvement intranslation quality.
sult was a modest 7% decrease in steps per sec-ond, but we were able to maintain the same model and batch sizes on P100 GPUs as Vaswani et al. (2017). 4 Experiments 4.1 Experimental Setup We use the tensor2tensor 1 library for training and evaluating our model. We evaluated our model on the WMT 2014 machine translation task, using the WMT 2014
Loading..
Information
Domain:
Source:
Link to this page:
Please notify us if you found a problem with this document: