Transcription of Self-Attention with Relative Position Representations
{{id}} {{{paragraph}}}
Example: biology
Self-Attention with Relative Position Representations
Self-Attention with Relative Position RepresentationsPeter UszkoreitGoogle VaswaniGoogle entirely on an attention mechanism,the Transformer introduced by Vaswani etal. (2017) achieves state-of-the-art results formachine translation. In contrast to recurrentand convolutional neural networks, it doesnot explicitly model Relative or absolute po-sition information in its ,it requires adding Representations of abso-lute positions to its this workwe present an alternative approach, extend-ing the Self-Attention mechanism to efficientlyconsider Representations of the Relative posi-tions, or distances between sequence the WMT 2014 English-to-German andEnglish-to-French translation tasks, this ap-proach yields improvements of BLEU BLEU over absolute Position representa-tions, respectively.
4.1 Experimental Setup We use the tensor2tensor 1 library for training and evaluating our model. We evaluated our model on the WMT 2014 machine translation task, using the WMT 2014 English-German dataset consisting of approxi-mately 4.5M sentence pairs and the 2014 WMT English-French dataset consisting of approxi-mately 36M sentence pairs.
Loading..
Tags:
Information
Domain:
Source:
Link to this page:
Please notify us if you found a problem with this document: