What does “Positional Encoding” mean?
Information added to token embeddings so transformers can distinguish sequence order.
How Positional Encoding Works
Token embeddings: no order info
Add position information
Model can tell 'dog bites man' from 'man bites dog'.
Practical Example
Without positions, a transformer would treat 'dog bites man' and 'man bites dog' as the same bag of words. Adding position information to each token tells it which came first.
Why It Matters
Word order carries meaning, and attention by itself ignores it. Different encoding methods affect how well models handle long texts.
Common Questions About Positional Encoding
Q: Original method?
The 2017 transformer used fixed sine and cosine patterns added to embeddings.
Q: Modern alternatives?
Learned positions and relative methods such as RoPE and ALiBi.
Q: Why does it affect context length?
Models can struggle with positions longer than those seen in training.