What is a token in AI? Singapore's four languages explain it

A chip motif standing for how a chatbot breaks text into pieces

Ask a chatbot “Have you eaten?” in English and it reads four things. Ask the same question in Tamil, using the tokenizer behind the original GPT-4, and it reads twenty-three.

Same question. Same politeness. Nearly six times the work. That gap is the quickest way we know to understand the one word every AI bill, every usage limit and every “context window” is measured in: the token.

A token is a piece, not a word

A chatbot does not read letters, and it does not quite read words either. Before your message reaches the model, a separate program called a tokenizer chops it into pieces from a fixed list it learned in advance, and swaps each piece for a number. The model only ever sees the numbers.

Google’s developer documentation puts it plainly: tokens “can be single characters like z or whole words like cat.” For English, its rule of thumb is that a token is about four characters, and “100 tokens is equal to about 60-80 English words.”

The analogy that holds best is a box of rubber stamps. The tokenizer has a stamp for every common chunk it saw often during setup: ” the”, ” you”, “berry”. A frequent word gets a stamp of its own. A rare word gets built from several smaller stamps. Something the stampmaker almost never saw gets spelled out in the tiniest stamps in the box, sometimes smaller than a single letter.

We ran a few words through OpenAI’s open-source tokenizer library, tiktoken, to see the stamps. “Unbelievable” comes out as three pieces: “un”, “belie”, “vable”. “Strawberry” is “str”, “aw”, “berry”. That second one explains an old party trick. When people ask a chatbot how many r’s are in strawberry, the model is not looking at ten letters. It is looking at three numbers, and has to have learned from somewhere how those pieces are spelled before it can count anything.

[[photo:1]]

One question, four official languages

Here is where Singapore makes the idea vivid. We took the most Singaporean question there is and counted the tokens in each official language, using two OpenAI tokenizers. The older one, called cl100k_base, is the one OpenAI’s own code maps to GPT-4 and GPT-3.5. The newer one, o200k_base, is mapped to GPT-4o, GPT-4.1 and GPT-5.

LanguageQuestionOlder tokenizerNewer tokenizer
EnglishHave you eaten?44
MalaySudah makan?64
Chinese你吃饭了吗?105
Tamilசாப்பிட்டீர்களா?237

Look at the Chinese row on the older tokenizer. Six characters became ten tokens, because the character for rice (饭) had no stamp of its own and was split into three fragments that are not even whole characters. Tamil fared worse: almost every piece was a fragment of a letter. Singlish, for what it is worth, travels well. “Can lah, no problem one.” is seven tokens, with ” lah” getting its own stamp.

Our test is four short phrases, so treat it as a demonstration, not a measurement. The measurement exists, though. A team at the University of Oxford, in a paper presented at NeurIPS 2023, ran the same 2,000 Wikipedia sentences, human-translated into 200 languages, through common tokenizers. On the GPT-4 tokenizer, Standard Malay needed 1.62 times as many tokens as English for the same content, Simplified Chinese 1.91 times, and Tamil 7.65 times. The worst case they found was Shan, a language of Myanmar, at more than 15 times.

[[photo:2]]

Why the count matters to anyone

Tokens are not a curiosity. They are the unit everything else is priced and rationed in.

The Oxford authors set out three consequences. Cost: services charge per token, so “users of some languages paying at least 2.5 times more for the same task as users of English.” Speed: “Some languages can require twice the time to process the same content as English.” And memory: a model can only hold a fixed number of tokens at once (its context window), so a conversation in a token-hungry language fills that window sooner. Google’s documentation confirms the billing side, saying the cost of a Gemini call “is determined in part by the number of input and output tokens.”

The same paper makes a point that is easy to miss. The unfairness “arises at the tokenization stage, well before the language model sees any data at all.” Before the model has a chance to be clever or clumsy in Tamil, the tokenizer has already decided how many pieces Tamil costs.

Why a Singapore team cared about the stamp box

This is not an abstract complaint here. AI Singapore, the national research network launched in 2017, builds SEA-LION, a family of open models covering 11 Southeast Asian languages including Malay, Tamil and Chinese, according to the team’s 2025 paper with NUS. When the team picked a base model to build on, the tokenizer was one of the reasons. “Gemma’s Tokenizer performs better when applied to the languages we have in our region,” William Tjhi, AI Singapore’s head of AI, told Google in its case study of the project.

And the big labs have moved too. Our own Tamil example dropped from 23 tokens to 7 between OpenAI’s older and newer tokenizers, so the gap the Oxford team measured in 2023 is narrower now for the newest models. Narrower is not gone. In our table, English still costs the least.

(A disclosure, since it touches us: Mentus AI runs mentors on large language models, so we pay per token too, and we have a commercial interest in people understanding how that works.)

What to do with this

Three habits follow once you know what a token is.

When a chatbot fumbles a spelling or letter-counting question, it is not being lazy. It never saw the letters. Asking it to write the word out one letter per line first hands it the letters, which is at least a fairer test.

Where a tool meters tokens, the language you write in changes how fast you spend them. A long Tamil or Chinese chat can use up a token allowance sooner than the same chat in English.

And when someone tells you a model “speaks” Tamil or Malay, the better question is how many pieces it needs to say a sentence. That number shapes the price, the speed and how much of the conversation it can remember. If you want to see the pieces for yourself, the tiktoken library is free and the experiment above takes about five minutes. Then read why ChatGPT makes things up, because once you see that the model is predicting one numbered piece at a time, the guessing habit starts to make sense.

Was this useful?

Sources

  1. Language Model Tokenizers Introduce Unfairness Between Languages (NeurIPS 2023) · Petrov, La Malfa, Torr and Bibi, University of Oxford (arXiv)
  2. Understand and count tokens · Google AI for Developers
  3. tiktoken: model-to-encoding mapping (model.py) · OpenAI (GitHub)
  4. SEA-LION: Southeast Asian Languages in One Network · AI Singapore and National University of Singapore (arXiv)
  5. AI Singapore makes AI more inclusive for Southeast Asia with Gemma 2 · Google DeepMind