Knowledge Maxxing← Back to topic library
AI MaxxingIntermediate75 sec
Positional encoding
Positional encoding is information added to each token so a transformer knows the order of elements in a sequence, since self-attention alone does not track position.
Prerequisite: Self-attention
What matters
Without it, a transformer would treat a sequence as an unordered set of tokens, unable to tell 'dog bites man' from 'man bites dog'.
Example
Positional encoding lets the model tell that 'not' appearing right before 'good' changes the meaning differently than 'not' appearing at the very start of the sentence.
Common misconception
Positional encoding is not optional bookkeeping; without it a transformer has no way at all to represent word order.
Recall question
Why does a transformer need positional encoding in addition to self-attention?
Keywords and related concepts
positional encodingword ordertransformerself-attention