See how GPT tokenizers turn text into tokens: a free tokenizer visualizer for GPT-4o, GPT-4 and GPT-2.
Language models don't read letters. They read tokens: numbers from a fixed vocabulary of
tens of thousands of text pieces. GPT models use byte-pair encoding (BPE). Training starts
from single bytes and keeps merging the most frequent adjacent pair into a new token, for example
t + he → the. Encoding replays those merges in the order they were
learned. Try it in the Toy BPE tabs above.
Paste it into the box above and the count updates as you type. As a rule of thumb, one token is about four characters, or three quarters of a word, of ordinary English. Code, numbers, emoji and other languages usually take more. You can also drop a text file onto the box to count all of its tokens.
Because it never sees the letters. In GPT-4o's tokenizer, " strawberry" is a single token
(ID 101830). The model receives that number and has to remember how the word is spelled instead of
reading it, which makes counting letters surprisingly error-prone.
Switch to Token IDs → Text above and paste the IDs in any format, for example
[13225, 2375]. You'll see each token and the decoded text. Pick the same encoding that produced
the IDs: the same number means different text in different vocabularies.
A Reddit username that became a single token in the GPT-2 and GPT-3 vocabulary but almost never appeared in the training text, so those models behaved strangely when asked to repeat it. Newer tokenizers such as o200k_base split it into ordinary pieces.
The space in front of a word is part of the token. "hello" is token 24912 in o200k_base, while
" hello" is 40617. Capitalization makes a difference too: "Hello" is 13225.
They are OpenAI's tokenizer vocabularies. o200k_base (about 200,000 tokens) is used by GPT-4o, GPT-4.1, the o-series and GPT-5. cl100k_base (about 100,000) is used by GPT-4, GPT-3.5 Turbo and text-embedding-3. gpt2 (about 50,000) is the original GPT-2 vocabulary. The same text gets different token IDs in each.
Tokenizers work on UTF-8 bytes. Common English words were frequent in the training data, so they became single tokens. A character outside ASCII takes 2 to 4 bytes, and rarer ones never got merged into a single token, so they split into several tokens. Some of those tokens are only part of a character.
No. Anthropic's Claude tokenizer isn't public, so it can't run in a browser. Anthropic's API has a token counting endpoint that returns only the count. This site covers OpenAI's open tokenizers.
No. Everything runs in your browser. There's no server, and nothing you type is uploaded or stored.