UTF-8, the standard way of encoding text for computers, uses variable-length byte sequences. ASCII characters (basic English letters) each use 1 byte. Cyrillic, Arabic, and most other non-Latin scripts use 2 bytes per character. CJK characters (Chinese, Japanese, Korean) use 3 bytes.

When a tokenizer hasn't seen a character sequence enough times to give it a dedicated vocabulary slot, it falls back to encoding that sequence byte by byte. For a Russian word using Cyrillic characters, this can mean 2 tokens per character rather than 1 token per word. The result is dramatic inflation — Russian text that means the same thing as an English sentence might cost 2–3× as many tokens.

Token costs directly translate to API costs, response length limits, and processing time. A Russian speaker using GPT-4 was, in effect, paying 2–3× more per meaningful exchange than an English speaker. This is an invisible structural inequity baked into the architecture.

Arabic presents several unique challenges for tokenizers trained primarily on Latin-script text. First, the script is right-to-left, which means word boundaries appear in reverse order. Second, Arabic letters change form depending on their position in a word (initial, medial, final, or isolated forms) — the same letter has up to four visual variants. Third, Arabic is highly inflected, with roots modified by prefixes and suffixes that compound complexity. Fourth, the Unicode encoding of Arabic uses multi-byte sequences for each character.

The result is that GPT-4's tokenizer, trained predominantly on English and Romance-language web text, treats Arabic as a largely unfamiliar sequence of byte patterns rather than a structured language with predictable morphology. The improvement in GPT-5 reflects deliberate effort to include more Arabic text in the vocabulary-building phase.

A further complication: Arabic has significant variation between Modern Standard Arabic (MSA) — used in writing, news, and formal contexts — and the many spoken dialects, which vary substantially by region. The sentence used here is MSA, the appropriate register for a written educational example, but a speaker is unlikely to use this form of the language for something as mundane as buying a train ticket. This register gap has no real equivalent in English, and tokenizers trained on internet text may handle formal and colloquial Arabic very differently. The translation used here was verified by Heather Sweetser, Senior Lecturer of Arabic at UNM.

This seems counterintuitive — Mandarin uses a completely different script (hanzi/Chinese characters) that requires 3 bytes per character in UTF-8, while Russian Cyrillic only needs 2 bytes. So why does Mandarin fare better?

The answer is training data composition. Mandarin Chinese has an enormous internet presence — a huge proportion of global web content is in Chinese. This means GPT-4's vocabulary-building phase saw enough Chinese character sequences to assign dedicated tokens to common characters. Russian Cyrillic, despite being used by hundreds of millions of people, had comparatively less representation in the training corpus.

This illustrates that tokenization efficiency is not primarily about linguistic or script complexity — it is about whose language was represented in the training data. It is a proxy for whose internet activity was included in the model's construction.

Standard written Japanese uses three scripts simultaneously in a single sentence. Kanji carry core semantic meaning; hiragana handle grammatical structure (particles, verb conjugations, etc.); katakana are used primarily for foreign loanwords and emphasis. A tokenizer must handle all three, plus the transitions between them.

In GPT-4, kanji tend to tokenize as individual characters (one token per character), while common hiragana sequences that appear frequently — like grammatical particles の, を, が, は — may get dedicated tokens due to their high frequency. This explains why Claude, which apparently has better hiragana coverage, achieves 22 tokens vs GPT-4's 28: the common grammatical endings are being recognized as units rather than byte sequences.

This also means that the tokenization "penalty" for Japanese varies significantly depending on the text content. Academic or technical text heavy in kanji tokenizes worse than casual conversational text heavy in hiragana.