What is a token? And why it is probably the most important concept for understanding AI

Several observations are called for.

First, a rare word like "Alyriaz" is split into several tokens (Aly + riaz), because the word did not exist when the model was trained. The model breaks it down into fragments it knows.

Second, frequent words like "est", "un", "qui" become a single token, because they appear millions of times in the training corpus. The model has "learned" them as a block.

Third, each token has a unique numerical identifier. This is what the model actually sees: sequences of numbers. Human text is merely the interface for us humans.

Fourth, spaces and punctuation count. A space before a word is part of the token. That is why you see " est" with a space in front of it.

Why tokens are not words

Here is a natural question: why not just use words? Why this complexity? Three important technical reasons.

Reason 1: handling unknown words. If the model only knew whole words, it would be unable to process proper nouns, neologisms, typos, trademarks, technical terms. With fragmented tokens, it can break down any text into elements it knows.

Reason 2: multilingual efficiency. Modern models speak dozens of languages. Rather than having a dictionary of words for each language (which would be gigantic), tokens allow a unified approach. Common fragments (prefixes, suffixes, roots) are shared. A model saves massively on memory thanks to this factorisation.

Reason 3: morphological consistency. In French, "manger", "mangé", "mangeais", "mangerons" are variations of the same verb. Tokens allow the model to learn that "mang" is a common root, and that the endings change according to tense. More efficient than memorising each form separately.

How AI "understands" tokens: the secret of context

Now that we know what a token is, how does AI make something of it? This is where the magic happens, and it is what radically distinguishes AI from classic search engines.

Each token is associated with a mathematical vector of several thousand dimensions. A vector, in mathematics, is a series of numbers representing a position in an abstract space. Imagine it as coordinates in a GPS, but with 4,096 dimensions instead of 2 (latitude/longitude). Each token has its "position" in this ultra-multidimensional space.

The point: tokens that are close in this space have close meanings. The vector for "cat" is close to that of "dog" (domestic animals), a little further from "lion" (wild feline), further still from "car" (inanimate object). The distances between tokens carry meaning. This is what researchers call embeddings (in French: plongements vectoriels).

When you type a question into ChatGPT, the model:

1. Splits your question into tokens
2. Converts each token into its vector
3. Calculates the relationships between all these vectors (this is the attention mechanism, which we documented in our article on SubQ and the end of the transformer)
4. Generates the most probable next token, given the whole context
5. Repeats the operation token after token until a complete answer is formed

It is technically dizzying, but conceptually simple: the AI predicts the most probable next token, again and again. When you talk to it about cooking, it predicts tokens consistent with cooking. When you talk to it about physics, tokens consistent with physics. The ability to maintain that consistency over thousands of tokens is what makes modern models impressive.

Why tokens limit what the AI can do

This explanation now makes it possible to understand several phenomena that can seem mysterious.

Why models have a limited "context window". Calculating attention over all the tokens in a text requires resources that grow (quadratically on classic transformers, as explained in our SubQ article). The longer the context, the more expensive and slow it is. This is why GPT-5 caps out at 200,000 tokens, Claude Opus 4.7 at 1 million tokens, Gemini 3 at 1 million as well. Beyond that, the cost becomes prohibitive.

Why the price is in tokens and not in words. Because that is the real technical cost. A request of 100 tokens costs 10 times less than one of 1,000 tokens. AI APIs bill what they actually spend, not an approximation in words.

Why "minority" languages cost more. The models were optimised on English, which dominates the internet. In English, the ratio is roughly 1 token = 0.75 words. In French, 1 token = 0.7 words. In Japanese, Arabic and Thai, the ratio can fall to 1 token = 0.3 words, because non-Latin characters are less well represented in the models' vocabulary. The consequence: writing in Japanese costs 2 to 3 times more than writing in English on ChatGPT. It is a little-known but real commercial bias.

Why hallucinations exist. The model predicts the most probable next token. "Most probable" does not mean "accurate" or "true". If several tokens are almost equivalent in probability, the model picks one at random, weighted. If the most probable continuation is plausible but incorrect (for example, citing a book that does not exist but that "sounds right"), the model hallucinates. This is intrinsic to how it works, not a bug.

Why models "forget" during a conversation. The context window is limited. If your conversation becomes too long (several hundred messages), the earliest messages fall outside the window and the model no longer sees them. It has no memory beyond the current context, unless the company adds an external memory to it (such as the Memory feature in ChatGPT introduced in 2024).

The detail that makes models drift: tokenisation biases

Here is a little-known but important point. The way a sentence is split into tokens influences what the model understands. And this tokenisation is not culturally neutral.

The vocabulary of models (the ~50,000 to 200,000 tokens they know) was built from the training corpus. Because the majority of the corpus is in English, common English words are single tokens, cheap to process. Common French words are also single tokens. But words in Arabic, Swahili or Mongolian are often split into several tokens, each of them infrequent.

Consequence: the model "understands" over-represented languages better. On scientific tests, GPT-5 achieves 92% success in English, 87% in French, 78% in Russian, 65% in Arabic, and drops drastically for less represented languages such as Wolof, Quechua or Tibetan.

This bias has been repeatedly denounced by independent researchers. It has real consequences: a student who learns in Swahili has access to a less competent AI than a student who learns in English. This is what researchers call "inequity by tokenization". As we explored in our article on Chinese models, models produced in China partly compensate for this bias for their own languages, but not for the rest of the world.

How to count tokens in your texts

To get a concrete idea, several free tools let you count the tokens of a given text. OpenAI offers Tokenizer on platform.openai.com/tokenizer, which shows exactly how GPT would split any text. Anthropic has a similar tool for Claude. Hugging Face (the largest hosting platform for open source models) offers "tokenizer comparators" that show how different models would split the same text.

These tools let you understand, for your own uses:

• If you want to summarise a book, how many tokens it represents, and therefore whether it fits in your model's context
• How much a session of use via API will cost you
• Which model is cheaper for your language (French consumes fewer tokens with Mistral than with OpenAI, for example)

Why recent progress revolves around tokens

Understanding tokens also helps to understand the current boundaries of AI and where research is heading.

The increase in context. Going from 4,000 tokens (GPT-3 in 2020) to 12 million tokens (SubQ in 2026) in 6 years represents a factor of 3,000. This is what enabled the shift from AIs that "forgot everything in 5 minutes" to AIs that can read entire books. This increase continues.

The fall in cost per token. In 2022, GPT-3 cost 60 dollars per million output tokens. In 2026, several equivalent models cost less than 1 dollar. This 60-fold drop in 4 years massively democratises access to AI. It continues.

Improving multilingual tokenisation. Several models (Mistral in France, Qwen in China, Llama 4 at Meta) are working on better representing non-English languages. As we explored in our article on France's position in AI, this is one of the areas of European innovation aimed at levelling costs and performance across languages.

Sub-quadratic architectures. As we documented in our article on SubQ, the major technical challenge is being able to process far more tokens without costs exploding. New architectures tackle this problem head-on.

Multimodal tokens. Modern models no longer process only text. An image is also converted into tokens (each image "patch" becomes a token). The same goes for voice and video. This is what allows Claude or Gemini to analyse an image you upload to them: they tokenise it and process it using the same mechanism as for text. The boundary between textual and visual becomes a boundary between types of tokens.

The practical application for you

Now that you understand tokens, how do you use this in your daily life?

To save on API costs. If you use AI professionally (via API rather than the web interface), count your tokens. A verbose 500-token prompt costs 5 times more than a precise 100-token prompt, often for equivalent quality. Conciseness pays.

To better manage long conversations. If you see that your AI is starting to "forget" elements mentioned at the beginning, it's because the conversation is exceeding its context window. Starting a new conversation with a summary of the important points is often better than continuing a session that is drifting.

To choose the right model. If you need to analyse a long document (a 100-page contract, a book, a codebase), choose a model with a large context (Claude Opus at 1M tokens, Gemini Flash at 1M tokens, SubQ at 12M tokens if validated). If you just need short, frequent answers, a model with 32k tokens is enough and costs less.

To understand the model's limitations. An AI that hallucinates a factual detail is not "failing": it is doing what its nature as a token predictor imposes on it. Systematically verifying important facts is mandatory, not optional. As we explored in our article on rigged demos, AI is not a source of truth, it is a probabilistic generation tool.

What you know now

You now know that:

• A token is a fragment of text (often half a word), not a complete word
• The AI does not see your words, it sees sequences of numerical identifiers associated with mathematical vectors
• It predicts the most likely next token, over and over, until it forms a response
• The context window is the maximum number of tokens it can process at once
• Less well-represented languages cost more tokens and are less well understood
• Recent progress revolves around larger context, lower costs, improved multilingual performance, and sub-quadratic architectures

You now have the basics to understand most technical AI news. When people talk about "200,000 tokens in Claude" or a "context window extended to 1 million", you know exactly what they mean. You know why some uses are expensive and others are not. You know why your AI hallucinates and why it does not remember your conversation from 3 weeks ago.

To go further, you can read our foundational article on how LLMs work, which covers the other essential concepts (training, inference, fine-tuning), and our analysis of SubQ and the end of the transformer, which shows how new architectures are shaking up these fundamentals. Technology becomes transparent when you understand its basic building blocks. And the basic building blocks of AI are tokens.