AI Foundations Playgrounds · Beginner
Tokenizer Playground
Type any text and watch it split into colour-coded tokens with IDs, counts and cost per model.
Live tokenization
Token IDs & counts
Cost comparison
Understanding Tokenization
What are Tokens?
Tokens are the smallest units of text that AI models understand. They can be words, parts of words, or even characters.
Each colored box represents one token
Example: "unhappy" → ["un", "happy"] = 2 tokens
Encoding Process
Text is converted into numerical IDs that models can process. Different models use different tokenizers.
Numbers show the unique ID for each token
Example: "hello" → Token ID: 15339
Cost Impact
API costs are calculated per token, not per word. Fewer tokens = lower costs and faster processing.
Token count directly affects your API bill
1000 tokens ≈ 750 words for most models
Tokenization Tips
- Shorter is cheaper: Concise text uses fewer tokens
- Common words: Frequently used words are usually single tokens
- Special characters: Punctuation and symbols can create separate tokens
- Different models: GPT, Claude, and other models tokenize text differently
Input Text
Example price tiers for learning — check your provider's pricing page for real rates.
Input Cost:
$0.000
Est. Output Cost (1K-token reply):
$0.000
Total Estimated Cost:
$0.000
Tokenized Output
0
Tokens
0
Characters
0
Words
0.0
Avg Token Length
Enter text above to see the tokenization magic happen