AI ART Discover · Learn · Create
C AI Glossary Term • Term 48 of 224
Plain English Definition

What does “Cross-Entropy Loss” mean?

Core Definition

A common objective function used to train classification models by penalizing incorrect probabilities.

Operational Mechanism

How Cross-Entropy Loss Works

1

Probability given to the right answerLoss (penalty)Cross-entropy lossExampleA classifier sees a photo of a dog. If it assigns 0.9 probability to 'dog', the loss is small

2

if it assigns only 0.1 to 'dog', the loss is large. Training adjusts the weights to push the probability on the correct class up.

Real-World Scenario

Practical Example

A classifier sees a photo of a dog. If it assigns 0.9 probability to 'dog', the loss is small; if it assigns only 0.1 to 'dog', the loss is large. Training adjusts the weights to push the probability on the correct class up.

Core Significance

Why It Matters

Cross-entropy is the standard training signal for classifiers and language models. It punishes confident wrong answers more than uncertain ones, which encourages sensible probabilities.

Expert Q&A

Common Questions About Cross-Entropy Loss

Q: Why not just count correct answers?

Counting is not smooth, so it gives training no direction. Cross-entropy changes gradually with the probabilities, which makes optimization possible.

Q: What does a loss of zero mean?

That the model gave probability 1 to the correct answer on every example, which on real data usually signals overfitting rather than perfection.

Q: Is cross-entropy used for language models?

Yes. Language models are trained to minimize cross-entropy on predicting the next token, and perplexity is derived from it.