AI ART Discover · Learn · Create
AI Art > Glossary > Mixture of Experts (MoE)
Browse Full AI Glossary
M AI Glossary Term • Term 129 of 224
Plain English Definition

What does “Mixture of Experts (MoE)” mean?

Core Definition

An architecture with specialized subnetworks where only selected experts run per input.

Operational Mechanism

How Mixture of Experts (MoE) Works

Mixture of Experts (MoE): Token comes in feed into Router picks the best 2 of 8 experts, which produces Expert 3 runs, Expert 7 runs.

Real-World Scenario

Practical Example

A model may hold many expert subnetworks but send each token to only two of them. This way, it has a very large total number of parameters while using only a fraction of them for each token.

Core Significance

Why It Matters

MoE lets models grow in capacity without a proportional rise in compute per token. It makes large models cheaper to run, though they still need lots of memory.

Expert Q&A

Common Questions About Mixture of Experts (MoE)

Q: Do experts specialize in topics?

Not necessarily in human-readable ways. They tend to specialize in patterns that help the model, which may not match neat subjects.

Q: What is the router?

A small network that scores the experts and decides which ones process each token.

Q: What is the trade-off?

All experts must be stored in memory, and balancing the load across them adds training complexity.