# What Is Tokenisation? The Invisible Mechanism Behind AI Costs, Quality, and Bias

Tokenisation drives 65% of unexpected AI costs. Learn how tokens work, why non-English content costs 2-7x more, and how to cut API spend by 30-50%.

**Published:** April 24, 2026
**Author:** Evan Ramzipoor

---

Two out of three IT leaders have been hit with unexpected AI charges. The culprit isn't bad software or hidden fees. It's tokenisation.

If you use AI tools or APIs, you're already paying for tokens whether you realize it or not.

**Key takeaways**

- Tokenisation splits text into subword pieces called tokens before AI can process it
- One token is roughly 0.75 English words or 4 characters; AI APIs charge per token, with output
tokens costing 3-10x more than input tokens
- Non-English languages require 2-7x more tokens than English, increasing cost and potential error
rates
- Concise prompts can cut token usage by 30-50%, and routing tasks to the right model can cut
costs by 50-75%
- While tokenisation-free models have begun to emerge, token economics will dominate production AI
for years

The Zylo AI Cost Report (2026) found that 65% of IT leaders report unexpected AI costs, with budgets running 30-50% over. The unit behind those charges is the token: the smallest piece of text an AI model reads, writes, or bills for. Tokenisation is the process by which text is broken into those pieces. It determines what you pay per API call, how much context your model can handle, and why ChatGPT can't count the R's in "strawberry."

## The Revenue Case: Why Tokenisation Is a Budget Problem

Tokenisation drives AI costs. Every prompt you send and every response you receive is billed per token. Output tokens (the text the model generates) cost 3-10x more than input tokens, which is why long responses drive surprise bills.

**Tokenisation drives AI costs**

Every prompt and response is billed per token. Output tokens cost 3-10x more than input.
- **65%** hit with surprise AI costs
- **99%** Token price drop: From $30/M tokens to $0.05/M since GPT-4 launch (IntuitionLabs, 2026)
- **75%** Smart routing savings: Teams routing simple tasks to smaller models cut API spend significantly
- **50%** Prompt optimisation: Basic prompt optimisation cuts token count by 30-50%
- **2.02T** Global AI spending forecast 2026: (IDC, up 37% from 2025)
- **65%** IT leaders with unexpected AI costs: (Zylo AI Cost Report, 2026)

Token prices have dropped 90-99% since GPT-4 launched, but costs still fluctuate based on language, content type, and output length. Teams that route simple tasks to smaller models cut spend by up to 75%. None of those savings are possible without understanding what a token actually is.

## What Is Tokenisation? The 60-Second Explainer

Tokenisation is the process by which AI breaks text into smaller pieces called tokens. A token can be a word, part of a word, or a character. Models only process tokens; they never see your original text as written.

Think of tokenisation like breaking a sentence into LEGO blocks. Some blocks are full words ("the," "cat"), while others are parts of words ("un" + "happi" + "ness"). OpenAI estimates one token is about 4 characters, or 0.75 English words. A 1,000-word English article is roughly 1,300 tokens.

**Three types of tokenisation**
- **Payment / Data Security**: Replaces sensitive data like credit card numbers with random placeholders for PCI DSS compliance and fraud prevention.
- **Blockchain / Assets**: Converts real-world assets (property, art, securities) into digital tokens on a blockchain for fractional ownership and trading.
- **AI / NLP (this article)**: Breaks text into subword units for language model processing. Determines cost, context limits, and output quality.

## How Tokenisation Works: From Text to Numbers

**How Tokenisation Converts Text to Numbers**
- **Split Text**: Step 1 A tokenizer breaks your text into small pieces. The word 'unhappiness' becomes three tokens: 'un', 'happi', and 'ness'. Common words like 'the' stay as single tokens.
- **Map to Numbers**: Step 2 Each token is mapped to a number from the model's vocabulary. 'un' might map to 348, 'happi' to 17487, and 'ness' to 655. The vocabulary size determines how efficiently text is encoded.
- **Process Numbers**: Step 3 The model processes these numerical sequences, not the original text. All reasoning, generation, and response creation happens in this numerical space. Output numbers are then mapped back to tokens and decoded into text.

### Tokenisation Methods

Modern AI models use one of three tokenisation approaches, each with different tradeoffs for cost and language coverage.

- **Byte Pair Encoding (BPE)**: The algorithm merges common characters until it builds a vocabulary of frequent patterns. BPE is the most widely used approach in AI today. It starts with single characters and merges frequent pairs iteratively. GPT-4's tokeniser has 100,256 tokens, while GPT-4o doubled that to roughly 200,000 for better multilingual performance. Used in GPT models
- **WordPiece**: Similar to BPE but picks merges based on likelihood rather than frequency. WordPiece evaluates which merge would most improve the model's ability to represent the training data, resulting in a vocabulary that is statistically optimised rather than frequency-optimised. Used in BERT and similar models
- **SentencePiece**: Treats text as raw bytes rather than pre-split words, which helps handle languages like Japanese or Thai that lack clear word boundaries. SentencePiece is language-agnostic by design, making it the preferred choice for multilingual AI models. Used in T5 and multilingual models

### The "Strawberry" Problem

Tokenisation helps explain many AI errors. Ask ChatGPT "how many R's are in strawberry?" and it'll probably get it wrong. The model doesn't see letters individually; it sees two tokens: "straw" and "berry." The R's are buried inside those chunks, so the model can't reliably count them. As AI researcher Andrej Karpathy noted, tokenisation is "probably the worst part of working with LLMs."

## Why Tokenisation Matters for AI Performance

Tokenisation shapes three important aspects of AI systems: what you pay, how well the model performs, and how fairly it treats different languages. For some languages, poor tokenisation can raise costs by 2-5x and cut accuracy by up to 18%.

### Maths, Reasoning, and the Token Gap

Tokenisation affects numbers too. When a model sees "1234," it might split that into "123" and "4," making basic arithmetic unreliable. Singh and Strouse ("Tokenisation Counts," arXiv, 2024) found that flipping token direction (right to left) boosted maths accuracy by more than 22%.

### The Token Tax: Language Bias Built Into AI

Research by Lundin et al. ("The Token Tax," arXiv, 2025) found that African languages need 2-5x more tokens than English to process the same content. In one example, a Telugu translation had fewer characters than the English version but needed 7x more tokens.

- **5x** More tokens for African languages: A 5x jump in tokens translates to roughly 25x higher costs for the same content
- **18%** Accuracy drop per extra token/word: Each additional token per word reduces LLM accuracy by 8-18% (Lundin et al., 2025)

### Context Rot: When More Tokens Make Things Worse

While large [context windows](https://www.searchable.com/glossary/context-window) might seem beneficial, Chroma Research (2025) tested 18 LLMs and found that performance drops as context grows, especially for information in the middle of a prompt. This is known as the "lost in the middle" effect. More tokens don't always improve results; they can do the opposite.

> Avoid feeding entire documents into context windows. Break content into targeted chunks and put
>   critical information at the start and end, not the middle.

## Token-Based Pricing: What You're Actually Paying For

- **3x** Output token cost multiplier (minimum): Output tokens cost 3-10x more than input tokens. Current input range: $0.50-$3/M tokens
- **200x** Price reduction since Jan 2024: LLM inference prices have fallen 50x-200x per year since January 2024 (Swfte AI, 2026)

As of early 2026, input tokens range from $0.50 to $3 per million, while output tokens cost $3 to $15 per million. To estimate your own costs, use OpenAI's tokeniser tool or this rule of thumb: multiply your English word count by 1.3. If you're managing AI spend, track the input-to-output token ratio. A ratio above 1:3 is a cost signal worth investigating.

## Practical Implications for Content and Marketing Teams

**Five ways to cut token costs**
- **Prompt Optimisation**: Short, clear prompts use 30-50% fewer tokens. Be specific, avoid repetition, use system prompts instead of repeating instructions.
- **Context Management**: Put critical info at start and end, not the middle. Break documents into chunks rather than feeding the whole thing.
- **Multilingual Budgeting**: Non-English workflows cost 2-7x more. A $500/month English workflow could cost $1,500-$3,500 in other languages.
- **Structured Content**: Clear headings, short paragraphs, and consistent word choices tokenise more efficiently and improve AI summarisation.
- **Model Routing**: Route simple tasks to cheaper models, complex ones to larger models. GPT-4o's 200K vocabulary handles non-English better than GPT-4's 100K.

## The Future of Tokenisation

Tokenisation may not last forever. Meta AI Research (December 2024) published the Byte Latent Transformer (BLT), which processes raw bytes and matches LLaMA 3's performance with up to 50% fewer compute operations. HKU NLP's EvaByte (2025) is the first open-source byte-level model that matches tokenised model performance.

If these approaches mature, the tokenisation pipeline could disappear. For now, AI runs on tokens, prices in tokens, and limits in tokens.

> Token-free models are promising but not yet production-ready at scale. Plan budgets around token
>   economics for at least the next 2-3 years.

## Frequently Asked Questions

### What is a token in ChatGPT?

A token is a piece of text, roughly 0.75 words or 4 characters of English. ChatGPT uses Byte Pair Encoding to split text into these pieces before processing. When OpenAI charges 'per token,' this is the unit. Common words like 'the' are 1 token, while longer words like 'understanding' might be split into 2 or 3.

### How many tokens is a word?

In English, one word is about 1.3 tokens. Short words ('the,' 'is,' 'at') are one token. Longer or rarer words get split into 2-4 tokens. Non-English words often need much more, with some languages using 2-7 times the tokens of English for the same content.

### Why is AI bad at maths and counting letters?

AI models see token chunks rather than single letters or digits. 'Strawberry' becomes ['straw,' 'berry'], so the model can't spot each R to count them. '1234' might become ['123,' '4'], which compromises maths.

### What is the difference between tokenisation in payments and AI?

Payment tokenisation swaps sensitive data (like card numbers) with safe placeholders. AI tokenisation chops text into small pieces so models can process it as numbers. Same word, completely different mechanisms.

### Does tokenisation affect AI output quality?

Yes. Each extra token per word reduces LLM accuracy by 8-18%, according to the Token Tax study (Lundin et al., 2025). Languages that require more tokens per word see higher error rates and lower quality output.

### How do I count tokens before sending a prompt?

Paste your text into OpenAI's tokenizer tool to see the exact breakdown. For code-based counting, OpenAI's tiktoken library works in Python. Quick rule: multiply your English word count by 1.3.

### Why does AI cost more for non-English languages?

Non-English text requires 2-7 times more tokens than English for the same content, so API costs go up. Most tokenizer word lists were built on English text, which means non-English words are less likely to be single tokens.

### What is BPE (Byte Pair Encoding)?

BPE is the tokenisation algorithm behind GPT models. It starts with single characters and merges common pairs until it reaches a target vocabulary size. If 'th' appears often in training data, it becomes one token.

## Wrapping Up

Tokenisation determines how much you pay, how much text a model can process, and how accurately it handles different languages. Understanding the mechanism turns unpredictable AI bills into something you can control.

- **Write short prompts**: Save 30-50% on token costs with concise, specific instructions.
- **Put key info at start and end of context**: The 'lost in the middle' effect means models perform worst on information in the centre of long prompts.
- **Pick models based on token efficiency**: Route tasks to the right model for your use case, not just the highest benchmark score.
- **Assume English pricing applies globally**: Non-English content costs 2-7x more in tokens. Budget accordingly.
- **Dump everything into context windows**: More tokens don't mean better results. Performance drops as context grows.
- **Ignore the input-to-output ratio**: Output tokens cost 3-10x more than input. A ratio above 1:3 is a cost signal worth investigating.

---

[Back to Blog](https://www.searchable.com/blog) | [Searchable Homepage](https://www.searchable.com)