AI ART Discover · Learn · Create
T AI Glossary Term • Term 204 of 224
Plain English Definition

What does “Tokenization” mean?

Core Definition

The process of splitting text into tokens for model input.

Operational Mechanism

How Tokenization Works

1

Raw text

2

Tokenizer applies learned splitting rules

3

Token IDs: 4021, 318, 257 ...

4

Fed into the model.

Real-World Scenario

Practical Example

Before a sentence reaches a language model, a tokenizer splits it into pieces and maps each to a number. The model only ever sees the numbers, and the output numbers are converted back to text.

Core Significance

Why It Matters

Tokenization affects cost, speed, and sometimes ability, such as with spelling, arithmetic, or non-English text.

Expert Q&A

Common Questions About Tokenization

Q: Do all models use the same tokenizer?

No. Each model family has its own, so token counts differ.

Q: How is the vocabulary decided?

Methods such as BPE learn common character sequences from a large text sample.

Q: Why do models struggle with letters in a word?

They see tokens rather than individual letters, which makes tasks like counting letters harder.