Knowledge MaxxingSelf-attention
Self-attention is the mechanism that lets a transformer weigh how relevant every other element in a sequence is to each element it is currently processing.
Prerequisite: Transformer
What matters
This weighting is what lets the model decide, for any given word, which other words in the sentence matter most for understanding it, rather than treating all context equally.
Example
In 'The trophy didn't fit in the suitcase because it was too big', self-attention lets the model weigh 'trophy' more heavily than 'suitcase' when resolving what 'it' refers to.
Common misconception
Self-attention does not process words strictly in order one at a time; it compares all positions in the sequence against each other simultaneously.
Recall question
What does self-attention let a model decide for each word in a sequence?