What Are AI Tokens? We Measured 10 Everyday Tasks to Find Out

TL;DR

  • A token is a small chunk of text, usually a whole word or part of one, that an AI model reads and writes. In English, one token is about four characters, or three-quarters of a word (OpenAI Help Center).
  • We ran 10 everyday tasks through OpenAI’s tokenizer. A simple “Hello” was 1 token; an 89-word work email was 107 tokens.
  • The same question cost twice as many tokens in Hindi as in English (18 vs 9), and a six-emoji birthday message used 13.
  • Tokens matter because AI companies charge per token, and every model can only hold a fixed number of them at once, called its context window.
  • Different AI companies count differently. Anthropic says its newer Claude models produce about 30% more tokens for the same text (Anthropic, November 2026).

AI tokens are the small pieces of text that chatbots like ChatGPT and Claude actually read, think in and charge for. A token can be a whole word, part of a word, a space, a number or even half an emoji. To find out what that means in practice, we counted the tokens in 10 everyday tasks, from saying hello to pasting a recipe.


How a token works, in plain English

Before an AI model can read your message, a program called a tokenizer chops it into pieces. Each piece is a token, and each token gets a number. The model never sees your words, only that list of numbers.

Common words usually become one token each. Longer or rarer words get split. In our tests with OpenAI’s tokenizer, “Hello” and “Mumbai” were one token each, “ChatGPT” was two (“Chat” + “GPT”), and “unbelievable” was three (“un” + “bel” + “ievable”).

OpenAI’s own rule of thumb is that 100 tokens is roughly 75 words of English, and a typical paragraph is about 100 tokens (OpenAI Help Center). It also notes that non-English text tends to use more tokens per character, which can make it more expensive to process (OpenAI Help Center).

Why split words at all? Because a model can’t keep a separate entry for every word in every language. Breaking text into reusable pieces lets it handle new words, typos and code. The trade-off is that the splits can look strange to humans. As AI researcher Andrej Karpathy put it in his tokenization lecture: “Note that all of these splits look completely arbitrary” (Andrej Karpathy, minbpe lecture).

He also warns that “Tokenization is at the heart of a lot of weirdness in LLMs” (Andrej Karpathy, minbpe lecture).

That’s why chatbots sometimes stumble on tasks that look easy to us, like counting the letters in a word: they see tokens, not letters.

One important catch: every company uses its own tokenizer. The same sentence can be a different number of tokens in ChatGPT, Claude or Gemini (if you’re still choosing between them, see our guide to ChatGPT alternatives in 2026). Anthropic, for example, says Claude 4.7 and later models use a newer tokenizer that produces about 30% more tokens for the same text (Anthropic pricing docs, November 2026).

We measured 10 everyday tasks

To make tokens less abstract, we wrote 10 short pieces of text you might send to a chatbot on a normal day. Then we counted their tokens with two of OpenAI’s tokenizers, using the open-source gpt-tokenizer library:

  • o200k_base, the tokenizer OpenAI lists for GPT-4o and GPT-4o mini (OpenAI Cookbook).
  • cl100k_base, the older tokenizer OpenAI lists for GPT-4 and GPT-3.5 Turbo (OpenAI Cookbook).

These counts cover only the text you type. They don’t include the hidden instructions an app adds, the model’s reply, or any “thinking” it does, all of which also use tokens.

The results

TaskWhat we typedWordsCharactersTokens (GPT-4o tokenizer)Tokens (GPT-4 tokenizer)
1. Say hello“Hello”1511
2. Ask a quick question“What’s the weather like in Mumbai today?”74099
3. Send a casual textA 15-word “running late” message with one emoji15671920
4. Write a short work emailAn 89-word email to a colleague89460107107
5. Post a tweet-length updateA 40-word mini review of a budget phone402144646
6. Ask in HindiThe weather question from task 2, in Hindi6261828
7. Paste a short Python functionAn 8-line function that averages numbers282065959
8. Send an emoji-heavy message“Happy birthday!!!” plus six emoji3251320
9. Share a link and contact detailsA booking URL, phone number and date101144141
10. List recipe ingredientsEight ingredients for a rice dish381876465

What the numbers tell us

Plain English is cheap. The email, the tweet and the quick question all landed at about 1.1 to 1.3 tokens per word, close to OpenAI’s “100 tokens is about 75 words” rule.

Other languages cost more. The Hindi version of our weather question used 18 tokens, twice the 9 tokens of the English one. On the older GPT-4 tokenizer it was 28, more than three times as many. In our test, the newer tokenizer narrowed the gap, but didn’t close it.

Emoji, links and numbers are expensive. Our three-word birthday message used 13 tokens because of its six emoji. A 10-“word” line with a web address and a phone number used 41 tokens, because URLs and digits get broken into lots of small pieces.

Code and lists sit in between. The Python function used about two tokens per word, and the ingredient list about 1.7. Symbols, indentation and line breaks all count.

The tokenizer matters. For everyday English, both of OpenAI’s tokenizers gave almost the same counts. The big differences showed up with Hindi and emoji, where the newer tokenizer used far fewer tokens.

Why token counts matter: price and memory

You pay by the token

If you use a free chatbot, you rarely see tokens. But developers and businesses using AI through an API (a direct connection to the model) pay for every token going in and every token coming out. Output tokens usually cost more than input tokens.

ModelInput price per 1M tokensOutput price per 1M tokensContext window
OpenAI GPT-5.5$5.00$30.001,050,000 tokens
Anthropic Claude Opus 5.5$4.00$20.001M tokens
Anthropic Claude Sonnet 5.5$2.00$10.001M tokens
Anthropic Claude Haiku 5.5 (prompts up to 100K tokens)$0.10$0.501M tokens

Sources: OpenAI GPT-5.5 model page, Anthropic pricing and Anthropic context windows, November 2026.

Those per-million prices sound big, but single messages are tiny. Our 107-token email would cost about $0.0005 to send to GPT-5.5 at its $5 input price, assuming a similar token count. That’s about a twentieth of a cent. Costs only add up at scale: a company sending millions of messages a day, or an AI agent reading long documents again and again.

Every model has a memory limit

The second reason tokens matter is the context window: the maximum number of tokens a model can work with in one go. Anthropic describes it as all the text a model can refer to when writing a response, including the response itself (Anthropic context windows, November 2026).

That window is shared. Your message, the earlier chat history, any files you upload, hidden system instructions, tool results and the model’s own “thinking” all count towards it (Anthropic context windows, November 2026). That’s why a long chat can suddenly feel forgetful or hit a limit, even if your latest message is short.

Bigger isn’t automatically better, either. Anthropic notes that accuracy and recall can drop as the token count grows, a problem it calls “context rot” (Anthropic context windows, November 2026). Vinit Mehta of the AI company Trail puts it simply: “Treat it like RAM, not a filing cabinet” (Trail, April 2026).

Trail’s guide also estimates that code uses roughly 1.5 to 2 tokens per word (Trail, April 2026). Our Python function came in at about 2.1, right at the top of that range.

Our verdict: your messages are cheap, the extras aren’t

For most people, tokens are not something to worry about day to day. A normal chat message is a few dozen tokens, and even a long email is around a hundred. What actually eats tokens is everything around your message: long chat histories, big file uploads, pasted web pages, and emoji- or link-heavy text.

Three habits make a real difference:

  • Start a new chat for a new topic, so the model isn’t carrying a long history it doesn’t need.
  • Paste only what’s needed, not a whole document, when you want help with one section.
  • Expect non-English text, code and links to cost more, and keep that in mind if you pay per token.

FAQ

1. How many AI tokens is one word?

In English, one word is about 1.3 tokens on average, because OpenAI estimates 100 tokens is roughly 75 words (OpenAI Help Center). Short, common words are usually one token, while long or rare words can be two or three.

2. Does ChatGPT count tokens the same way as Claude?

No. Each company uses its own tokenizer, so the same text can give different counts. Anthropic says its newer Claude models produce about 30% more tokens for the same text than earlier ones (Anthropic pricing docs, November 2026).

3. Why do other languages use more tokens?

Tokenizers learn their pieces from the text they were built on, so words that show up often tend to get their own tokens, while words in less common languages get split into more pieces. In our test, a Hindi question used twice as many tokens as the same question in English.

4. What happens when you run out of tokens?

If a request is too long for the model’s context window, it fails or the reply gets cut off. On Anthropic’s API, an over-long prompt returns a “prompt is too long” error (Anthropic context windows, November 2026). Chat apps may instead summarise or drop older messages, or tell you that you’ve hit a usage limit.

5. How can I count tokens myself?

OpenAI offers a free online tokenizer at platform.openai.com/tokenizer, and developers can use its open-source tiktoken library (OpenAI Cookbook). Anthropic offers a token-counting API so you can check a request before sending it (Anthropic context windows).