What does “BPE (Byte Pair Encoding)” mean?
A subword tokenization algorithm that merges the most frequent character pairs into reusable tokens.
How BPE (Byte Pair Encoding) Works
Start with characters: l o w e r
Merge the most frequent pair: l+o
Repeat merges: lo, low, ...
Vocabulary of common pieces.
Practical Example
In a training corpus where 'low' and 'lower' are common, the pair 'l','o' merges first, then 'lo','w', and so on. Common words become single tokens, and rare words are broken into familiar pieces.
Why It Matters
BPE balances vocabulary size against sequence length and avoids unknown words. Most large language models use a variant of it.
Common Questions About BPE (Byte Pair Encoding)
Q: Why subwords?
They handle rare and new words by splitting them into known parts.
Q: Does BPE work on bytes?
Byte-level BPE starts from bytes, so any text can be represented.
Q: Where did it come from?
It began as a data-compression method and was adapted for translation models in 2015.