AI ART Discover · Learn · Create
T AI Glossary Term • Term 203 of 224
Plain English Definition

What does “Token” mean?

Core Definition

A chunk of text processed by language models, such as a word piece or symbol.

Operational Mechanism

How Token Works

1

Text: 'Unbelievable!'

2

Split into pieces

3

Un | believ | able | !

4

Each piece gets a number ID.

Real-World Scenario

Practical Example

The word 'unbelievable' might be split into three tokens, while a short common word like 'the' is one. Providers count tokens to limit input and to price usage.

Core Significance

Why It Matters

Tokens determine cost, speed, and how much fits in the context window. Different languages can need different token counts for the same meaning.

Expert Q&A

Common Questions About Token

Q: How many words is a token?

In English, a token averages roughly three-quarters of a word, but it varies by tokenizer and language.

Q: Why not use whole words?

Pieces handle rare words and new spellings by combining familiar parts.

Q: Are spaces and punctuation tokens?

They are often included in or alongside tokens.