TechnicalAI glossary
Tokenizer
The program that splits text into tokens before a model reads it, using fragments learned according to how frequent they are. Its effects are very concrete: switching tokenizer can make the same text consume noticeably more tokens (and therefore cost more), French is often split into more pieces than English, and numbers get chopped up with no regard for their value.
In other words
The guillotine cutter that slices a page into strips before handing it over to be read: depending on the blade, the same page yields more or fewer pieces.
Alyriaz articles that mention it
- The model rankings as of 23 September 2026
State of the model landscape as of 23 September 2026: who leads on what, at what price, and how to read leaderboards that have become hard to compare.
- The languages the big models ignore
State-of-the-art models perform poorly in most of the world's languages. We explain why, what it costs hundreds of millions of people, and why local players are stepping in.
- It's happening today: the Sonnet 5 price increase takes effect, and it's steeper than 50%
The promotional rate for Claude Sonnet 5 ends today. We explain why the real increase exceeds the advertised 50%, and what you should concretely check in your costs.
- Why would an AI capable of solving a mathematical conjecture get two numbers wrong when adding them?
AI models excel at advanced mathematics yet sometimes fail at basic arithmetic. Here's why: they don't calculate—they predict text, and numbers are tokenised in a way that makes arithmetic surprisingly difficult.
- On 1 September, Claude Sonnet 5 increases by 50%. And there's a catch almost no one mentions.
The promotional rate for Claude Sonnet 5 ends on 31 August 2026: from 1 September, input and output costs rise by 50%. But the real trap lies elsewhere: the new tokenizer counts more tokens for the same text. We explain the actual calculation and how to prepare now.
- Why does your French cost more than English when you use an AI?
For equivalent content, writing in French consumes more tokens than English, which increases cost and reduces the available space in the context window. We explain why this inequality exists, what it changes in practice, and how to limit it.