Knowledge MaxxingAttention mechanism
An attention mechanism lets a model directly weigh the relevance of every other element in a sequence to the one it is currently processing, rather than passing information step by step.
Prerequisite: Recurrent neural network
What matters
Because it connects distant elements directly instead of through a chain of intermediate steps, attention avoids the long-range information loss that recurrent networks are prone to.
Example
An attention mechanism can let a model relate a pronoun near the end of a long paragraph directly back to the noun it refers to near the beginning.
Common misconception
An attention mechanism is not unique to text; it is used across many data types, including images and audio, wherever relating distant elements directly is useful.
Recall question
Why does attention avoid the long-range information loss recurrent networks are prone to?