Knowledge MaxxingKnowledge distillation
Knowledge distillation trains a smaller student model to reproduce the behavior of a larger teacher model, aiming to keep most of its capability at lower cost.
Prerequisite: Large language model
What matters
Because the student learns from the teacher's outputs rather than raw data alone, it can often reach performance closer to the teacher's than training the same small model from scratch would.
Example
A large model's outputs on thousands of examples are used to train a much smaller model, which then runs faster and cheaper while retaining much of the original's accuracy.
Common misconception
Knowledge distillation does not usually produce a student that fully matches the teacher's accuracy; some capability is typically lost in exchange for the smaller size.
Recall question
What does the student model learn from in knowledge distillation, beyond raw training data?