Knowledge MaxxingRecurrent neural network
A recurrent neural network processes a sequence one element at a time, carrying a hidden state forward from each step to the next to retain information about earlier elements.
Prerequisite: Convolutional neural network
What matters
Because information must pass step by step through the hidden state, recurrent networks can struggle to preserve details from far earlier in a long sequence, motivating later architectures like attention.
Example
A recurrent network reading a sentence word by word updates its hidden state at each word, carrying forward a summary of everything read so far.
Common misconception
A recurrent neural network is not obsolete for every use case; for short sequences or resource-constrained settings, it can still be a reasonable, cheaper choice than a transformer.
Recall question
Why can a recurrent network struggle to remember information from early in a long sequence?