What does “Gradient Descent” mean?
An optimization method that updates parameters in the direction that reduces error.
How Gradient Descent Works
Start with random weights
Measure the loss
Compute the gradient
Step downhill by the learning rate
back to the start.
Practical Example
Picture standing on a foggy hillside and taking small steps in whichever direction slopes downward most steeply. Gradient descent does this with the model's loss, adjusting weights a little each step until the loss stops dropping.
Why It Matters
Gradient descent and its variants are how nearly all neural networks are trained. Step size and data handling choices affect whether training succeeds.
Common Questions About Gradient Descent
Q: What is stochastic gradient descent?
A version that estimates the gradient from a small random batch of data each step, which is faster and works well in practice.
Q: Can it get stuck?
It can slow near flat regions or settle in a local minimum, though in large networks these are usually less troublesome than expected.
Q: What is Adam?
A popular variant that adapts step sizes per parameter using running averages of gradients.