AI ART Discover · Learn · Create
G AI Glossary Term • Term 91 of 224
Plain English Definition

What does “Gradient Descent” mean?

Core Definition

An optimization method that updates parameters in the direction that reduces error.

Operational Mechanism

How Gradient Descent Works

1

Start with random weights

2

Measure the loss

3

Compute the gradient

4

Step downhill by the learning rate

5

back to the start.

Real-World Scenario

Practical Example

Picture standing on a foggy hillside and taking small steps in whichever direction slopes downward most steeply. Gradient descent does this with the model's loss, adjusting weights a little each step until the loss stops dropping.

Core Significance

Why It Matters

Gradient descent and its variants are how nearly all neural networks are trained. Step size and data handling choices affect whether training succeeds.

Expert Q&A

Common Questions About Gradient Descent

Q: What is stochastic gradient descent?

A version that estimates the gradient from a small random batch of data each step, which is faster and works well in practice.

Q: Can it get stuck?

It can slow near flat regions or settle in a local minimum, though in large networks these are usually less troublesome than expected.

Q: What is Adam?

A popular variant that adapts step sizes per parameter using running averages of gradients.