AI ART Discover · Learn · Create
S AI Glossary Term • Term 185 of 224
Plain English Definition

What does “Scaling Law” mean?

Core Definition

An empirical relationship showing how performance improves with model size, data, or compute.

Operational Mechanism

How Scaling Law Works

Compute, data, or model sizeLoss (lower is better)Loss falls smoothly with scaleExampleResearchers trained many models of different sizes and found that test loss decreased in a predictable pattern as size, data, and compute increased. That let them estimate the results of a bigger run before paying for it.

Real-World Scenario

Practical Example

Researchers trained many models of different sizes and found that test loss decreased in a predictable pattern as size, data, and compute increased. That let them estimate the results of a bigger run before paying for it.

Core Significance

Why It Matters

Scaling laws guide how labs spend training budgets. They are empirical trends, not guarantees, and they may not hold forever.

Expert Q&A

Common Questions About Scaling Law

Q: Who found them?

Studies from OpenAI in 2020 and DeepMind's 2022 'Chinchilla' work are among the best-known.

Q: Does bigger always win?

Within a regime, loss tends to improve, but returns diminish and data quality matters.

Q: What does Chinchilla say?

That, for a given compute budget, many large models were too big for the amount of data used, and balancing the two works better.