What is the "tokenizer tax"?
The tokenizer tax refers to an issue of pricing disparity for different languages for LLM API providers. This is a consequence of the fact that LLM API pricing is usually per token, but model tokenizers treat different languages differently. Some languages get "chopped up" much more than others, and specifically, more than English, which most models and tokenizers are often optimized for. As a result, it is much more expensive to query LLMs in those languages as compared to in English.
This tax can be quantified using parallel text, which has the same content translated into various languages. For a given language , we can take sentence and its translation into language , its translation () into English, and compare the number of tokens yielded. We can average this ratio over many parallel sentences from our corpus , and compute the mean token ratio:
The higher this mean token ratio for a language, the costlier it is to use the LLM in it. Studies find that is high for many languages, especially low-resource ones [2][3][4]; for example, up to x for Malayalam with some GPT models [4].
Background: Comparing tokenization across languages
The goal of tokenization is to break a piece of text into smaller reusable units for which we can learn embeddings. A lower number of tokens for a given text is not always better. For example, we don't want an entire sentence in English to be a single token: it wouldn't get used very often. It's somewhat clear that very fine chopping is bad, for example, splitting apart diacritics from letters. This is because token sequences get longer, and LLMs are worse at longer sequences. Also, the LLM now has to first "detokenize" this text internally before it can understand its semantics. However, we still want some chopping, which is the point of tokenization. We actually still do not really know how to describe the goodness of a tokenization scheme, or what the ideal amount of chopping is, from the perspective of performance [5].
A concept we use to characterize tokenization is token fertility, or the number of tokens per "word" in a language. There is work in reducing this for languages that get very chopped up, by for example, allocating enough space in the tokenizer vocabulary to some language. But for the same reason as above, lower token fertility is not always better: there's a sweet spot. Crucially, this sweet spot varies widely across languages, because words are very different creatures in different language. For example "Gelemeyecekmişsiniz" in Turkish means "It seems you won't be able to come". So a space-separated word in Turkish typically should have more tokens than a space-separated English word.
Similarly, characters are also different creatures in different scripts / languages. Characters in scripts that are syllabaries, such as Hiragana, include a consonant and a vowel, whereas logographic scripts like the Mandarin script, a single character carries semantic meaning. The same holds for the number of bytes used to represent a piece of text, which again depends not only on the script of the language, but the script encoding used.
This is to say that in general, we cannot use simple text length metrics for making cross-lingual comparisons of tokenization: we cannot compare the number of tokens yielded per word or character of text across languages, or claim that it should be equal.
Should mean token ratio be 1? In theory and in practice?
Let's think about a cross-lingual quantity of text over which the number of tokens should be equal. Let's understand meaning or content as the thing that some piece of text is trying to convey, and information as the bits of communication used to convey that content. Note that content is quite comparable across languages: it is what is transferred when we translate a piece of text.
Intuitively, the same amount of information, or bits of communication, should ideally be expressed in the same number of tokens. The correspondence between a token and a unit of information is natural: tokenization is an encoding, and encoding length depends on the number of bits in the utterance.
Mean token ratio is computed over equivalent sentences, or in other words, equivalent amounts of content. If a single Turkish word generally carries more content than an English, there should be proportionately fewer words needed to express the same thing, and we should roughly end up with the same number of tokens.
And so, can and should be 1? Maybe achieving is one ideal of multilingual tokenization, but it may not be perfect either in theory or practice, and that's for two reasons.
Firstly, content and information may not exactly correspond. While content is indeed preserved across languages for parallel sentences by definition, the bits of information used may not be. In order to express the same thing, different languages may grammatically or pragmatically require different things to be specified. For example, Hindi present tense verbs require the specification of the gender of the subject, whereas Chinese present tense verbs don't. Some languages always need you to specify how you know what you know (like Tuyuca). Others require an indication of the relationship between people in a dialog (like Japanese). Even if there is no grammatical requirement, pragmatic or usage norms in a language may require specification of some extra things in one language that are not required in another (think politeness norms, or think about concepts used very often in one language but requiring explication in another). These things all require bits of information.
Note that the goal of an expert translation is to convey the same content, not the same pieces of information. So even if content exactly corresponds, does the information transmitted exactly correspond for any two languages? Do languages use different levels of specification? Do some languages just... say more? Even if information transmitted over identical content corresponds on average across utterances for two languages, does it really for each equivalent utterance? These are genuine questions, to which computational linguistics may have interesting responses. In my hobbyist linguist capacity, however, I would say that the answer is no: content and information do not exactly correspond across languages.
An important side note here: in spoken language, the ratio of bits/second transmitted is very stable across languages [6]. Languages that compress more into single syllables are spoken slower, and languages that use more syllables for the same thing are usually spoken faster. "Information" in this study is used to mean the reduction in uncertainty per syllable in a monolingual context: this definition is very related (although obviously technically different) from our use of the term information (but not content). In other words, this study says that languages transmit information at the spoken rate, but makes no claim about the amount of information needed to convey the same content.
All this is to say that, if you agree with me that ideal tokenization dictates that a token should encode the same number of bits of information across languages, and given that is computed over equivalent amounts of content, it's not clear even in theory that the ideal mean token ratio should be 1.
Next: even if the information expressed in parallel sentences were indeed exactly the same across languages, considerations of model design, script, and linguistic typology might still dictate that mean token ratio be different from 1. Tokenizers are engineering components rather than scientific objects. Maybe the LLM just does much better when a piece of text expressing something in Swahili is one big chunk whereas in Turkish it's fifteen. Script and typology play a role here: some languages are agglutinative, with neatly separated units of meaning; other languages are fusional, and separation is not so easy. Text simply looks different in different languages, and tokenizer and model design should be free to optimize performance rather than reflect scientific truths. In other words, it's not really necessary from a practical engineering perspective that mean token ratio be 1.
Good tokenizers are not necessarily fair under token-based pricing
This leads us to the following point: the question of designing good multilingual tokenizers, in my opinion, can and should be decoupled from the question of charging fair prices. Reducing tokenizer fertility, or determining the sweet spot for choppingness, is not a matter of pricing fairness: it's a matter of model performance. A model with high for Marwadi may very well be a better multilingual model, depending on what good tokenization looks like for Marwadi versus for English. But the price of that model for English and for Marwadi should still be the same for the same service.
Obviously, the reason we talk about token fertility-type metrics and pricing in the same breath is because of token-based pricing. But let's not conflate these issues. Let's talk about token-based pricing.
Economics and fairness of token-based pricing
Token-based pricing makes sense on the part of the model developer. The number of FLOPs needed to process the input in a function of the number of tokens in that input, and so the fact that Marathi costs more than English is simply a reflection of the fact that LLMs for Marathi are a more expensive product to provide than for English, given the state of technology today. There is no incentive for API providers to pay the extra cost out of their pocket: the consumer foots it. Like for any expensive product, the cost can be driven down if someone develops cheaper LLMs for Marathi, and Marathi users flock to that system, incentivizing other companies to make their own systems cheaper to run for Marathi.
Obviously, this is still unfair. Many essays about the history and socioeconomics of language inequity, the violence of linguistic colonization, self-perpetuating language-based injustice, and its reflection in technology can be inserted here. Whatever the logic of the market is, language communities are not bound to accept its terms. It is still valid to demand high-quality and equitably priced language technology as a matter of social language-based justice. Achieving quality parity seems like a long battle, but achieving pricing parity is less so.
What should a user be charged for?
Proposition: Two users asking for the same thing in different languages should be charged identically. We should charge users per unit content.
We should not charge users for information, for length of the input text by any text length metric, or for the token count of their utterance. Hopefully you are now convinced that all these things are controlled by factors such as linguistic structure, script, and model design. None of these is in the hands of the user, and they should not be penalized or favored for them. What is in their hands is how much they are asking for. This is comparable cross-lingually, and this is what we should care about.
Implementing content-based pricing
Recall that parallel text provides us with translations of the same content across several languages. Recall mean token ratio , computed for the particular tokenizer under question. This number precisely tells us the average ratio of tokens used to express the same content in language as compared to English under the tokenization scheme .
Proposition: For a given tokenizer , a token in should be times cheaper than a token in English (or more expensive, as the case may be). That is, if is the cost of an English token, then the adjusted cost per token in language should be
This lets API providers continue to determine pricing per token for English, and to investigate better multilingual tokenization from a performance perspective, while language communities pay fair prices.
Research and engineering directions towards content-based pricing
We need to figure out some things in order to make this work.
Language identification (LID): The above pricing model requires us to know the language of the input. So it needs a fast and cheap LID step over each input. This can run in parallel and won't be slower than the actual LLM run.
Dialects, code-switching, very short texts, and other LID thorns: LID people will rightly tell you how LID is not solved for the 7000 dialects and languages of the world. Language is quirky, dialects are ill-defined, texts may be too short, and orthography is not standard. For practical purposes, I'm not concerned: if we group all languages into a few hundreds of language clusters and show the approximate acceptability of using the same price adjustment within a cluster, we'll do fine from the perspective of classification accuracy.
What happens when inputs are language mixed and script mixed? Some heuristics or policy strategies can apply here. We can pick a single "majority" one if the ramifications can be shown to be acceptable, or we could try to actually classify each token given its entire context.
Estimating : Do we really know that the translations in the parallel text used to compute correspond to the same content across both sides? Are we dependent too much on the quality of the parallel text? Is stable across domains and registers?
My opinion: generally we can and do widely trust that content corresponds across translations even if information does not (by definition of translation as a task), and we should of course design the parallel text collection over which is computed to be diverse and representative. If we still doubt the quality or diversity of existing parallel data, then nothing stops us from creating a resource targeted towards this purpose. More parallel data is only better.
These are interesting research and engineering bumps which future work should look at. Nothing that we can't figure out!
There's also perhaps the following more open-ended issue. Note that even in a future where linguistic power is balanced, a high-quality multilingual model within our current paradigm of LLMs could still be more expensive to run for some languages, simply because need not be 1 for equally good downstream performance. Someone has to pay for processing those extra tokens. Whether this should be the speaker of the language or the service provider becomes a matter of policy and politics, as well as of driving R&D in a particular direction: it may increase the pressure on research and engineering to develop tokenization schemes that jointly optimize downstream performance with lower . Research is thinking about this already, e.g. [7]. Maybe content-based pricing will force industry to think a little too.
Conclusion
The world of tokenization is hairy and fun. The world of inflated prices for the same (usually worse) service for various languages is less fun. Let's do content-based pricing for LLMs: a user should pay the same for asking the same thing, no matter what language they ask in.
References
- The Tokenizer Tax. Coalition for Language Equity in Research (CLEAR).
- Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. 2023. Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. ↩
- Jessica M. Lundin, Ada Zhang, Nihal Karim, Hamza Louzan, Guohao Wei, David Ifeoluwa Adelani, and Cody Carroll. 2026. The Token Tax: Systematic Bias in Multilingual Tokenization. In Proceedings of the 7th Workshop on African Natural Language Processing (AfricaNLP 2026).↩
- Priyansh Srivastava. 2026. The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages. arXiv preprint arXiv:2607.24276. ↩
- Antonio Lopardo, Luis Frentzen Salim, Shikhar Srivastava, Kevin Jiang, Mridul Sharma, and Catherine Arnett. 2026. No Single Tokenizer Feature Reliably Predicts Downstream Language Model Performance. In Findings of the Association for Computational Linguistics: EMNLP 2026, ↩
- Christophe Coupé, Yoon Mi Oh, Dan Dediu, and François Pellegrino. 2019. Different languages, similar encoding efficiency: Comparable information rates across the human communicative niche. Science advances. ↩
- Negar Foroutan, Clara Meister, Debjit Paul, Joel Niklaus, Sina Ahmadi, Antoine Bosselut, and Rico Sennrich. 2026. Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics .↩