AI ART Discover · Learn · Create
AI Art > Glossary > Multi-Head Attention
Browse Full AI Glossary
M AI Glossary Term • Term 134 of 224
Plain English Definition

What does “Multi-Head Attention” mean?

Core Definition

A transformer mechanism that runs several attention operations in parallel to capture different types of relationships.

Operational Mechanism

How Multi-Head Attention Works

Multi-Head Attention: Same input tokens feed into Several attention heads in parallel, which produces Head 1: grammar links, Head 2: reference links, Head 3: nearby words.

Real-World Scenario

Practical Example

In one layer, one head may track which noun a pronoun refers to, while another focuses on neighboring words. Their results are combined into a richer representation.

Core Significance

Why It Matters

Multiple heads let the model look at different relationships at once. Heads tend to specialize, though not always in neat human terms.

Expert Q&A

Common Questions About Multi-Head Attention

Q: How many heads?

It varies, from a handful to a hundred or more in large models.

Q: How are head outputs combined?

They are concatenated and passed through a learned linear layer.

Q: Does each head really specialize?

Some do, but many are redundant, and interpreting them is an active research area.