Two out of three IT leaders have been hit with unexpected AI charges. The culprit isn't bad software or hidden fees. It's tokenisation.
If you use AI tools or APIs, you're already paying for tokens whether you realize it or not.
Key Takeaways
Tokenisation splits text into subword pieces called tokens before AI can process it
One token is roughly 0.75 English words or 4 characters; AI APIs charge per token, with output tokens costing 3-10x more than input tokens
Non-English languages require 2-7x more tokens than English, increasing cost and potential error rates
Concise prompts can cut token usage by 30-50%, and routing tasks to the right model can cut costs by 50-75%
While tokenisation-free models have begun to emerge, token economics will dominate production AI for years
The Zylo AI Cost Report (2026) found that 65% of IT leaders report unexpected AI costs, with budgets running 30-50% over. The unit behind those charges is the token: the smallest piece of text an AI model reads, writes, or bills for. Tokenisation is the process by which text is broken into those pieces. It determines what you pay per API call, how much context your model can handle, and why ChatGPT can't count the R's in "strawberry."
The Revenue Case: Why Tokenisation Is a Budget Problem
Tokenisation drives AI costs. Every prompt you send and every response you receive is billed per token. Output tokens (the text the model generates) cost 3-10x more than input tokens, which is why long responses drive surprise bills.
Tokenisation drives AI costs
Every prompt and response is billed per token. Output tokens cost 3-10x more than input.
From $30/M tokens to $0.05/M since GPT-4 launch (IntuitionLabs, 2026)
Teams routing simple tasks to smaller models cut API spend significantly
Basic prompt optimisation cuts token count by 30-50%
Token prices have dropped 90-99% since GPT-4 launched, but costs still fluctuate based on language, content type, and output length. Teams that route simple tasks to smaller models cut spend by up to 75%. None of those savings are possible without understanding what a token actually is.
What Is Tokenisation? The 60-Second Explainer
Tokenisation is the process by which AI breaks text into smaller pieces called tokens. A token can be a word, part of a word, or a character. Models only process tokens; they never see your original text as written.
Think of tokenisation like breaking a sentence into LEGO blocks. Some blocks are full words ("the," "cat"), while others are parts of words ("un" + "happi" + "ness"). OpenAI estimates one token is about 4 characters, or 0.75 English words. A 1,000-word English article is roughly 1,300 tokens.
Three types of tokenisation
Payment / Data Security
Replaces sensitive data like credit card numbers with random placeholders for PCI DSS compliance and fraud prevention.
Blockchain / Assets
Converts real-world assets (property, art, securities) into digital tokens on a blockchain for fractional ownership and trading.
AI / NLP (this article)
Breaks text into subword units for language model processing. Determines cost, context limits, and output quality.
How Tokenisation Works: From Text to Numbers
How Tokenisation Converts Text to Numbers
A tokenizer breaks your text into small pieces. The word 'unhappiness' becomes three tokens: 'un', 'happi', and 'ness'. Common words like 'the' stay as single tokens.
Tokenisation Methods
Modern AI models use one of three tokenisation approaches, each with different tradeoffs for cost and language coverage.
Byte Pair Encoding (BPE)
Used in GPT modelsThe algorithm merges common characters until it builds a vocabulary of frequent patterns. BPE is the most widely used approach in AI today. It starts with single characters and merges frequent pairs iteratively. GPT-4's tokeniser has 100,256 tokens, while GPT-4o doubled that to roughly 200,000 for better multilingual performance.
“If 'th' appears frequently in training data, it becomes one token. Common words like 'the' become single tokens, reducing overall token counts and costs.”
The "Strawberry" Problem
Tokenisation helps explain many AI errors. Ask ChatGPT "how many R's are in strawberry?" and it'll probably get it wrong. The model doesn't see letters individually; it sees two tokens: "straw" and "berry." The R's are buried inside those chunks, so the model can't reliably count them. As AI researcher Andrej Karpathy noted, tokenisation is "probably the worst part of working with LLMs."
Understanding How AI Processes Your Content
Searchable tracks how AI engines retrieve, process, and cite your content across platforms. See where your brand stands.
Why Tokenisation Matters for AI Performance
Tokenisation shapes three important aspects of AI systems: what you pay, how well the model performs, and how fairly it treats different languages. For some languages, poor tokenisation can raise costs by 2-5x and cut accuracy by up to 18%.
Maths, Reasoning, and the Token Gap
Tokenisation affects numbers too. When a model sees "1234," it might split that into "123" and "4," making basic arithmetic unreliable. Singh and Strouse ("Tokenisation Counts," arXiv, 2024) found that flipping token direction (right to left) boosted maths accuracy by more than 22%.
The Token Tax: Language Bias Built Into AI
Research by Lundin et al. ("The Token Tax," arXiv, 2025) found that African languages need 2-5x more tokens than English to process the same content. In one example, a Telugu translation had fewer characters than the English version but needed 7x more tokens.
More tokens for African languages
5x
A 5x jump in tokens translates to roughly 25x higher costs for the same content
Accuracy drop per extra token/word
18%
Each additional token per word reduces LLM accuracy by 8-18% (Lundin et al., 2025)
Context Rot: When More Tokens Make Things Worse
While large context windows might seem beneficial, Chroma Research (2025) tested 18 LLMs and found that performance drops as context grows, especially for information in the middle of a prompt. This is known as the "lost in the middle" effect. More tokens don't always improve results; they can do the opposite.
Avoid feeding entire documents into context windows. Break content into targeted chunks and put critical information at the start and end, not the middle.
Token-Based Pricing: What You're Actually Paying For
Output token cost multiplier (minimum)
3x
Output tokens cost 3-10x more than input tokens. Current input range: $0.50-$3/M tokens
Price reduction since Jan 2024
200x
LLM inference prices have fallen 50x-200x per year since January 2024 (Swfte AI, 2026)
As of early 2026, input tokens range from $0.50 to $3 per million, while output tokens cost $3 to $15 per million. To estimate your own costs, use OpenAI's tokeniser tool or this rule of thumb: multiply your English word count by 1.3. If you're managing AI spend, track the input-to-output token ratio. A ratio above 1:3 is a cost signal worth investigating.
Practical Implications for Content and Marketing Teams
Five ways to cut token costs
Prompt Optimisation
Short, clear prompts use 30-50% fewer tokens. Be specific, avoid repetition, use system prompts instead of repeating instructions.
Context Management
Put critical info at start and end, not the middle. Break documents into chunks rather than feeding the whole thing.
Multilingual Budgeting
Non-English workflows cost 2-7x more. A $500/month English workflow could cost $1,500-$3,500 in other languages.
Structured Content
Clear headings, short paragraphs, and consistent word choices tokenise more efficiently and improve AI summarisation.
Model Routing
Route simple tasks to cheaper models, complex ones to larger models. GPT-4o's 200K vocabulary handles non-English better than GPT-4's 100K.
The Future of Tokenisation
Tokenisation may not last forever. Meta AI Research (December 2024) published the Byte Latent Transformer (BLT), which processes raw bytes and matches LLaMA 3's performance with up to 50% fewer compute operations. HKU NLP's EvaByte (2025) is the first open-source byte-level model that matches tokenised model performance.
If these approaches mature, the tokenisation pipeline could disappear. For now, AI runs on tokens, prices in tokens, and limits in tokens.
Token-free models are promising but not yet production-ready at scale. Plan budgets around token economics for at least the next 2-3 years.
Frequently Asked Questions
A token is a piece of text, roughly 0.75 words or 4 characters of English. ChatGPT uses Byte Pair Encoding to split text into these pieces before processing. When OpenAI charges 'per token,' this is the unit. Common words like 'the' are 1 token, while longer words like 'understanding' might be split into 2 or 3.
Wrapping Up
Tokenisation determines how much you pay, how much text a model can process, and how accurately it handles different languages. Understanding the mechanism turns unpredictable AI bills into something you can control.
Write short prompts
Save 30-50% on token costs with concise, specific instructions.
Put key info at start and end of context
The 'lost in the middle' effect means models perform worst on information in the centre of long prompts.
Pick models based on token efficiency
Route tasks to the right model for your use case, not just the highest benchmark score.
Assume English pricing applies globally
Non-English content costs 2-7x more in tokens. Budget accordingly.
Dump everything into context windows
More tokens don't mean better results. Performance drops as context grows.
Ignore the input-to-output ratio
Output tokens cost 3-10x more than input. A ratio above 1:3 is a cost signal worth investigating.
See How AI Engines Process and Cite Your Content
Searchable monitors your visibility across ChatGPT, Perplexity, Gemini, and Google AI Overviews. Understand how your content gets sourced.
