Knowledge Maxxing← Back to topic library
ML MaxxingIntermediate75 sec
Learning rate
The learning rate controls how large a step gradient descent takes when adjusting a model's parameters at each iteration.
Prerequisite: Gradient descent
What matters
Too high a learning rate can cause training to overshoot and never settle; too low a rate can make training so slow it barely improves within a reasonable time.
Example
A learning rate set too high might cause a model's loss to swing wildly between iterations instead of steadily decreasing.
Common misconception
A smaller learning rate is not always safer or better; too small a rate can make training impractically slow without necessarily improving the final result.
Recall question
What can happen to training if the learning rate is set too high?
Keywords and related concepts
learning ratestep sizegradient descenthyperparameter