What does “RoPE (Rotary Positional Embeddings)” mean?
A positional encoding method that rotates query and key vectors to encode relative token positions.
How RoPE (Rotary Positional Embeddings) Works
Query and key vectors
Rotate by an angle that depends on position
Dot product reflects relative distance
Attention knows how far apart tokens are.
Practical Example
Two tokens three places apart get rotated by amounts whose difference depends only on that gap. Attention can then favor nearby tokens in a way that works wherever in the text the pair appears.
Why It Matters
RoPE is used in many popular open models and supports techniques that extend context length. It encodes relative position directly in attention.
Common Questions About RoPE (Rotary Positional Embeddings)
Q: Who proposed RoPE?
Su and colleagues in the 2021 RoFormer paper.
Q: Can RoPE models handle longer text than they trained on?
Not automatically, but scaling tricks can extend usable context, often with some fine-tuning.
Q: RoPE vs absolute encoding?
Absolute encoding marks each token's place; RoPE encodes how far apart tokens are.