What Are Tokens in LLMs? Tokenization Explained

What exactly is a token?

LLMs don’t read letters or words — they read integers. A tokenizer chops text into chunks from a fixed vocabulary (often 50,000 to 200,000+ entries) and maps each chunk to an ID. Common words are usually a single token; rare or long words are split into pieces.

Notice the leading spaces: in most tokenizers the space belongs to the next word, so " cat" and "cat" are different tokens. Exact splits vary by model.

How tokenization works (byte-pair encoding)

Most modern LLMs use a variant of byte-pair encoding (BPE) or a similar sub-word method such as SentencePiece:

1. Start with individual bytes or characters as the vocabulary.
2. Scan a large training corpus and find the most frequent adjacent pair (e.g., t + h).
3. Merge it into a new token (th) and repeat — thousands of times — until the vocabulary hits its target size.

The result: frequent strings (“ the”, “ing”, “ function”) become single tokens, while anything rare is spelled out from smaller pieces. Because it falls back to bytes, a BPE tokenizer can represent any text, including emoji and every language — just not equally efficiently.

Live token & cost estimator

Rules of thumb: converting tokens to words

Tokens and the context window

The context window is the model’s working memory for one request, measured in tokens. Everything shares it:

• the system prompt and instructions,
• the conversation history,
• retrieved documents (in RAG) and tool results (in agents),
• and the output the model generates.

Today’s windows range from tens of thousands to over a million tokens depending on the model. Bigger isn’t automatically better: long prompts cost more, slow responses down, and models can overlook details buried in the middle of very long inputs. Retrieval that sends only the relevant chunks usually beats pasting everything.

How token pricing works

API providers charge per million tokens, with separate prices for input and output. Output tokens typically cost several times more than input tokens because they’re generated one at a time. The formula:

cost per call = (input_tokens  × input_price  / 1,000,000)
              + (output_tokens × output_price / 1,000,000)

monthly cost  = cost per call × calls per day × 30

Worked example (illustrative prices): a support assistant sends 2,500 input tokens (instructions + retrieved articles + question) and gets 400 output tokens back. At $3 / 1M input and $15 / 1M output, each call costs $0.0075 + $0.006 = $0.0135. At 20,000 calls a day that’s about $8,100 a month — and trimming retrieved context from 2,000 to 1,000 tokens saves roughly a fifth of it. Plug in your own numbers with the LLM Cost Estimator.

Many providers also discount cached input (a repeated prompt prefix such as long instructions) and batch jobs that don’t need an immediate response. Reasoning models may also generate hidden “thinking” tokens that are billed as output.

Token quirks that explain “weird” LLM behaviour

• Letter counting: asking how many r’s are in “strawberry” is hard because the model sees a couple of tokens, not ten letters.
• Arithmetic on long numbers: numbers split into uneven chunks, which makes digit-by-digit maths error-prone — use a calculator or code tool.
• Word limits drift: the model plans in tokens, not words, so “exactly 100 words” is approximate.
• Trailing spaces matter: a prompt ending in a space can produce odd continuations because the space normally belongs to the next token.

7 ways to use fewer tokens (and spend less)

1. Send only the relevant chunks, not whole documents — tune chunk size and top-k.
2. Summarise or truncate long chat histories.
3. Keep stable instructions at the start of the prompt so prompt caching can apply.
4. Ask for concise output and set a max-tokens limit.
5. Use structured output instead of verbose prose when the result feeds code.
6. Route simple requests to smaller, cheaper models.
7. Use batch processing for non-urgent jobs where available.