Knowledge MaxxingLong-context model
A long-context model is designed to process very large context windows, often spanning hundreds of thousands of tokens, without requiring the input to be chunked.
Prerequisite: Context window
What matters
Achieving this requires specialized techniques, since attention's cost grows with sequence length; without them, processing very long input becomes too slow or expensive to be practical.
Example
A long-context model might take an entire novel as input in one request and answer questions that depend on details from both the first and last chapters.
Common misconception
A long context window does not guarantee the model uses distant information equally well; models can still underuse content buried in the middle of very long input.
Recall question
Why can't attention simply be applied to very long sequences without any special technique?