What does “Mixture of Experts (MoE)” mean?
An architecture with specialized subnetworks where only selected experts run per input.
How Mixture of Experts (MoE) Works
Mixture of Experts (MoE): Token comes in feed into Router picks the best 2 of 8 experts, which produces Expert 3 runs, Expert 7 runs.
Practical Example
A model may hold many expert subnetworks but send each token to only two of them. This way, it has a very large total number of parameters while using only a fraction of them for each token.
Why It Matters
MoE lets models grow in capacity without a proportional rise in compute per token. It makes large models cheaper to run, though they still need lots of memory.
Common Questions About Mixture of Experts (MoE)
Q: Do experts specialize in topics?
Not necessarily in human-readable ways. They tend to specialize in patterns that help the model, which may not match neat subjects.
Q: What is the router?
A small network that scores the experts and decides which ones process each token.
Q: What is the trade-off?
All experts must be stored in memory, and balancing the load across them adds training complexity.