Transcription of Self-Attention with Relative Position Representations
{{id}} {{{paragraph}}}
Example: biology
Self-Attention with Relative Position Representations
Self-Attention with Relative Position RepresentationsPeter UszkoreitGoogle VaswaniGoogle entirely on an attention mechanism,the Transformer introduced by Vaswani etal. (2017) achieves state-of-the-art results formachine translation. In contrast to recurrentand convolutional neural networks, it doesnot explicitly model Relative or absolute po-sition information in its ,it requires adding Representations of abso-lute positions to its this workwe present an alternative approach, extend-ing the Self-Attention mechanism to efficientlyconsider Representations of the Relative posi-tions, or distances between sequence the WMT 2014 English-to-German andEnglish-to-French translation tasks, this ap-proach yields improvements of BLEU BLEU over absolute Position representa-tions, respectively.
Position encodings based on sinusoids of vary-ing frequency are added to encoder and decoder input elements prior to the first layer. In contrast to learned, absolute position representations, the authors hypothesized that sinusoidal position en-codings would help the model to generalize to se-quence lengths unseen during training by allowing
Loading..
Tags:
Information
Domain:
Source:
Link to this page:
Please notify us if you found a problem with this document: