Deep Learning Dictionary - Lightweight Crash Course

Adaptive Learning Rates - Deep Learning Dictionary

In neural network training, the learning rate is a fractional value, typically set between 0.01 and 0.0001, for which the gradients are multiplied by before updating the weights with this resulting product. This value is a hyperparameter that we set prior to training.

The concept of adaptive learning rates allows us to make use of dynamically changing learning rates during training, but without the hassle of defining a learning rate schedule.