What does “SentencePiece” mean?
A language-agnostic tokenizer that learns subword units directly from raw text without pre-splitting on whitespace.
How SentencePiece Works
Raw text, spaces included
Learn pieces from the data
Treat space as a symbol
Works for any language.
Practical Example
For languages such as Japanese or Thai that do not separate words with spaces, SentencePiece learns its pieces directly from the text without needing a word-splitting step first.
Why It Matters
A language-agnostic tokenizer simplifies multilingual models. It is used in several well-known models.
Common Questions About SentencePiece
Q: Who created it?
Taku Kudo and John Richardson at Google, in 2018.
Q: Which algorithms does it support?
BPE and a unigram language-model method.
Q: SentencePiece vs BPE?
BPE is an algorithm; SentencePiece is a toolkit that can use BPE or other algorithms on raw text.