⚡ Quick Answer

In AI, tokenization refers to the process of splitting input text into individual units, or tokens (which can include words, characters or even parts of words), and assigning each token a unique ID that the AI model can use to process it. For example, an LLM would process “hello world” as [15339, 1917], two integers which represent the words in the input text in the fixed vocabulary of the model. As the model processes input, generates output, etc. it is always processing numeric sequences of IDs, and not actual text.

What Tokenization actually does

All Large language models are foundation models trained on text. In order to learn statistical relationships between sentences, words, and even parts of words, text needs to be converted into a fixed-size vocabulary of discrete units which the model can assign weights and probabilities to. Tokenization is the step in which raw text is converted into such a vocabulary of tokens.

While there are different ways to perform tokenization, the process itself is relatively sequential and follows a particular order or pipeline regardless of the root technology. First, raw text is split into tokens by a tokenizer. Then each token is looked up in a fixed vocabulary and converted into a fixed-size integer ID. This ID is then mapped to an embedding vector, which is the actual numeric representation that the model itself computes. When the model generates its own sequence of output token IDs, these IDs are detokenized by a detokenizer back into actual text. Note that all of the “real work” in the model happens between tokenization and detokenization, and that the tokenization and detokenization stages themselves are, themselves, trivial and simply part of the input and output processing of the model.

Note that in Web3 / payments, “tokenization” is something very different: representing real-world assets as digital tokens on a blockchain (e.g. bonds, real estate), or masking sensitive payment card / account numbers to prevent fraud (so-called “tokenization” for fraud prevention). None of this has anything to do with how an LLM processes text.

What Are the Main Types of Tokenization?

Four methods can be used to split text into individual parts. Each of them has its strengths and weaknesses. In general, word tokenization is simple to use and to implement. However, it does not work well for contractions and for words that consist of multiple words (so-called compound words). Subword tokenization splits up words into their subwords (smallest units of language). This method is the most popular for large language models. The main trade-off is between the size of the vocabulary and the length of the resulting sequences of tokens.

Word level tokenization – splitting on whitespace. This is simple, but tends to fail for things like contractions and compound words. It requires a separate vocabulary entry for each word form, so “run”, “runs”, “running” and “ran” would each require their own slot in the vocabulary for example. That can very quickly grow to hundreds of thousands of entries for even a moderate sized vocabulary.

Character tokenization is used when the units of text such as words or spaces do not exist or are not relevant for the specific task such as languages without word boundaries like Chinese or for tasks like handwriting recognition where the character is the fundamental unit of text. With character tokenization, the vocabulary is very small, typically less than 200 tokens for a given alphabet, but the sequence length is very large.

Subword tokenization: WordPiece (BERT), Unigram/SentencePiece (T5/LLaMA). The ‘industry standard’ is to use subword tokenization, i.e. to split rare words into their subwords (e.g. incompetence -> in-competence). A model trained on such tokens can then recognize words it has not seen during training by recognizing the subwords that make up the word. This approach is typically implemented using byte pair encoding (BPE), which is described in more detail below. OpenAI’s GPT models use BPE variants through the tiktoken library. The cl100k_base encoding used by more recent GPT-4 models has a vocabulary of roughly 100,000 tokens. Google’s SentencePiece library implements a related approach called Unigram tokenization, where a very large candidate vocabulary is first created and then pruned in order to remove tokens that contribute the least to the overall likelihood of the corpus. The remaining tokens then make up the vocabulary of the model.

Morphological tokenization is the process of breaking words into their constituent morphemes. A simple example would be to break the word “incompetence” into the three tokens “in-”, “competent” and “-ence”. The morphological tokenizer splits the input text at the real linguistic boundaries between the words, as opposed to simple character or subword tokenization which splits the input at artificial boundaries. The morphological tokenizer requires significant knowledge of language-specific rules, and is thus much more difficult to scale up to support a large variety of languages than the simple statistical approaches outlined above.

Vocabulary size, context windows, and why token count is a cost line item 

vocabulary size is a cost line item for large language models. It is set before training the model. Typically, for modern LLMs, it can range from 30,000 to 100,000+ tokens (or even more) and can be set for example for 50,000

As a consequence of a fixed vocabulary size (commonly in the order of 30,000 to 100,000+ tokens for modern LLMs), a design choice has to be made. A larger vocabulary size results in fewer tokens being needed to represent a sentence, hence resulting in shorter sequences of tokens that can be processed faster by the model. However, the larger vocabulary size results in a larger embedding table and in more parameters that are needed to represent the vocabulary. A smaller vocabulary size results in longer sequences of tokens that are needed to represent a sentence of a given length, but the model is much leaner.

This, in turn, means that the length of a sequence in tokens (not characters or words) is a key characteristic of an LLM’s context window (i.e. how much text it can process in one call to the model). For English text, the number of tokens in a given piece of text averages out at around 1 token per 4 characters, or 0.75 tokens per word (although this will vary from tokenizer to tokenizer and from language to language). And, most importantly, almost every commercial LLM API charges by the number of tokens (input plus output) processed by the model. This means that what seems like a short piece of text in words can actually be very token expensive if it contains rare words, code or is written in a language other than English and has been tokenized by a BPE vocabulary trained on mostly English text.

Most tokenizers reserve a small number of special tokens to use for model-specific purposes, like tracking sequence structure. For example, BERT-style models use [CLS] and [SEP] to denote start of sequence and between two segments respectively. For generative models, models use <bos> and <eos> to denote start and end of sequence respectively. A <PAD> token is also typically included to fill out sequences that are shorter than a fixed batch length, and an <UNK> token is often included to denote tokens not in the vocabulary, though this is less common in subword tokenized models where most inputs can be broken down into the subword pieces already in the vocabulary.

Why this one step matters for everything downstream

The real value in subword tokenization is for the model to be able to reasonably handle words it has never seen before. This includes proper nouns (like brand names), misspelled words, and words from domains that the model is not trained on (such as very technical text). Tokenization for languages that do not separate words with whitespace (such as Chinese), or that have very heavy inflection (such as Russian or Arabic), use other forms of tokenization, but these are the exception rather than the rule. Every other NLP task is applied on top of tokenization (of some form), so a poor choice of tokenization for the corpus at hand will affect the accuracy of intent, entity, sentiment, and other text classification tasks. This degradation in performance will typically not have an obvious cause.

Frequently Asked Questions

Tokenization is the process of taking in text and splitting it up into the individual tokens (words, characters, etc) and then converting those tokens into numbers that can be processed by a computer.

This is to say that a token can be more than just a single word. For example, a number, an abbreviation, a person’s name, or even a single character can be treated as a single token by a tokenizer.

LLMs use tokenization to read human language and turn it into numbers the model can then use to read in and write out text.

Common methods used to split text into tokens are word tokenization, character tokenization, subword tokenization and morphological tokenization.

No. The number of tokens that a single word will be split into can vary depending on a number of factors including the tokenizer, the size of the vocabulary, the language, and even the word itself.

Tokenization is the process of splitting text into tokens (words, characters, etc.) and mapping them to token IDs. The embedding then converts the tokens into numerical vectors (such as Word Embeddings) that attempt to capture some of the semantic or contextual meaning in the text.

Tokenization is how generative AI models process human language. The model uses the tokenized text to read in what you’ve written and to create written responses.