Machine Learning · Chapter 30 of 40
Gradient Descent
GRADIENT DESCENT minimizes a loss by taking steps in the direction of the negative gradient.
Learning rate controls step size — too big overshoots, too small is slow.
Example 1 (python)
# Pseudocode:
# for i in range(iters):
# grad = compute_gradient(w)
# w = w - lr * gradThe core optimization loop.
Example 2 (python)
# Stochastic GD updates per sample; mini-batch uses small batchesSGD scales to huge datasets.
Key points
- Takes steps down the loss surface.
- Learning rate = step size.
- Batch, mini-batch, and stochastic variants.
- Foundation of deep learning.
💡 Note: Adam is a popular adaptive optimizer that usually beats plain SGD for deep networks.