What is an AI Token?

What Is an AI Token?

A token is a unit of information for an AI model when processing text. The human mind perceives language as a chain of words, whereas an AI processes text as individual tokens. These are generated by the model’s tokenizer, the process of turning input text into a format that can be processed by a model. 

The size of a token can vary greatly depending on the word. Common words are usually represented by one token only but less common words can be split up into several tokens, such as parts of words or even characters. Punctuation, numbers, whitespace and special characters all affect the way in which text is tokenized.

By defining tokens differently than words, AI systems can count up the information in terms of tokens to determine the processing requirements. Also, when you input a prompt to an AI system, the input prompt is converted to tokens and processed by the AI model. The output generated by the AI model is a sequence of tokens too, which are then converted back to human-readable text.

So, tokens are a central element in modern language models, the amount of information a model can process, and thus also the cost of an API request to process a prompt. Also, for applications that process long conversations or even entire documents, tokens are a key element.

How Does Tokenization Work?

There is more to Tokenization than meets the eye. To begin, we feed a sentence to the input of an AI model. The first part (a so-called tokenizer) processes the input – be it a sentence or a string of characters – in order to arrive at the so-called tokens which the model can then process and in turn output.

The output of tokenization is a numerical representation of the input tokens. When a language model processes the tokens, it actually processes a numerical representation of the tokens.

Tokenization is not fixed across models. A given model may use a specific tokenizer and vocabulary to go from text to numbers. For that reason, same text can have different numbers of tokens depending on the model. That is also why for purposes of estimating context usage, or cost of API calls, it is necessary to use the tokenizer of the specific model that is going to process the text.

How Text Is Broken Into Tokens

Words in text can consist of complete words and parts of words and also of characters like spaces, punctuation and even numbers. But as already mentioned, modern language models use subword-based tokenization, which means that even common words will be split up into so-called subwords (also called wordpieces) and then again processed by the model, one after the other.

For example, a common word might be in the vocabulary, but less common words, including technical terms and even names, will typically be split into subwords. The model can then process each subword individually, greatly increasing the number of words that the model can handle.

In addition, such a subword approach helps dealing with so-called “unknown words”, which a model has not seen before during training. For example, proper names, recently coined terms and words with slightly different spellings can all be represented using the model’s subwords.

Why One Word Does Not Always Equal One Token

One word might equal one token, but it generally does not. More accurately, one word can represent a single token, but also many tokens or even part of another token depending on the surrounding words and even characters.

The final count depends on various factors, such as the specific tokenizer used, the language, the model’s vocabulary, the specific spelling or punctuation used, and even the content type (normal text vs. source code vs. equations in text etc.).

Therefore, it is only possible to make an approximate estimate of word-to-token ratios. In the end, a tokenization process using a specific model has to be executed to give an accurate number of tokens for a specific document.

What Are the Different Types of Tokens?

The categories of tokens may vary depending on the method used for the tokenization. However, in general, tokens are words (so-called word tokens), subwords (so-called subword tokens), characters (so-called character tokens) and even punctuation marks (so-called punctuation tokens) or other groups of characters.

Word Tokens

Word tokens are tokens which consist of one word. These words are often in the vocabulary of the model, hence suitable for word level modeling of languages with large amounts of very common words.

However, creating a vocabulary to store every possible word would result in a very large vocabulary. Moreover, for words not included in the vocabulary, such as new words, variations in spelling, words in specialized domains, and names, the model would not be able to handle input containing such words.

These models generally use a more flexible method for their language.

Subword Tokens

Subword tokens represent parts of words such as word beginnings or parts in the middle. By splitting long words into their individual parts, models can use the same patterns in many words to represent them.

This method is especially useful for so-called “technical language”. The number of possible subwords of a word can be much larger than the number of possible tokens. Thus, the vocabulary of a model can be much smaller than with word tokens.

Subword tokenization is one of the approaches in natural language processing to trade off between vocabulary size and expressing language. 

Character and Punctuation Tokens

Some tokenization systems can even look at individual characters. Punctuation and special characters are often also included as tokens.

This becomes especially relevant for models written for programming code, structured data, mathematical notation and technical documents. Often even short documents are composed of a large number of tokens due to special symbols and the appropriate formatting of the text.

How Are Tokens Used in AI Models?

Every step of AI’s processes for generating and processing language are completed through the use of tokens. When you input a prompt to an AI, for example, it first has to be translated into tokens that can be processed by the AI model. From there, it processes the input tokens in the order they were received to try to understand the scope of the available context for its response.

In addition to processing the input tokens, the AI model can also generate output tokens which can be sent back to the user.

Token processing is fundamental to the way that large language models work. Even though you experience writing an entire paragraph, for example, the model is in fact writing it token by token.

Input Tokens and Output Tokens

For models using a fixed size context (input) the number of input tokens, the context size, will also need to be specified. This could be for example a user’s prompt, system instructions, previous conversation, retrieved documents, examples etc.

Output tokens represent the generated output of the model.

For example, long prompts are counted in input tokens, while long responses are counted in output tokens. So when designing an application around text-based AI interactions, it is very important to take both into consideration.

To illustrate this, consider an AI assistant for an application that requires the user to refer to a large document every time they issue a request to the AI. In contrast to an assistant that only retrieves for the assistant the relevant parts of previous documents issued by the user as prompt, in this scenario the AI would be using a huge number of input tokens every time the user issues a request to it.

Tokens in Large Language Models

This is to say that large language models process language as sequences of tokens. Once tokens have been turned into text, that text is converted into numbers that can be processed by the model.

The model then generates tokens one by one. It predicts the next token in the sequence based on the context it has. It keeps generating tokens until it reaches the end of the sequence or a maximum output length.

In general, the way a language model processes words and sentences to eventually generate new text is token-by-token. And while that new text consists of words and sentences, it was generated by the model processing individual tokens.

Why Do Tokens Matter?

Tokens also matter as they influence a range of different issues, including the limits of context, the computational power required, costs of using models through APIs, and how models process language in order to generate output.

To illustrate, a user can generally remain unaware of the difference between words and tokens. However, it is of material significance to the AI application developer and his supporting AI development team when processing large numbers of documents, extended conversations and very high volumes of requests for service.

Tokens and Context Windows

The context window of a model can range from a few tokens to millions of them. Typically, the context window consists of the user’s prompt and all previous messages of a conversation. Additionally, a model might also consider system messages, input retrieved from the web, or other information that the application using the model provides to it.

The more information an AI application uses for the context, the more token capacity it needs. Therefore, the information has to be managed carefully in long documents as well as in extended conversations. Often there is much unnecessary information in those cases.

That’s why we use summarization and retrieval in AI applications. Instead of sending the entire knowledge base to the model every time it’s requested, only the relevant parts are sent as smaller sections of information.

Tokens and AI Model Costs

The number of input and output tokens processed via APIs is also often used by AI providers and services to calculate their usage and associated costs.

A short prompt and response would typically use very few tokens, whereas a prompt with a large document and very long response would use many more.

These issues can impact the cost of the API calls, especially when scaling large amounts of users. These token issues can be mitigated by minimizing unnecessary tokens (i.e. reducing context, removing repetitive prompts, limiting retrieved information, etc.) and setting reasonable limits for generated responses.

This does not mean we want to use the fewest tokens possible to keep costs down. Our goal with an AI application is to give the AI’s model enough information to handle the request and complete the task to our satisfaction.

Tokens and Model Performance

The amount of information in the tokens will have an impact on application performance. Too much information can result in increased processing, and too much context can lead to information overload, meaning the model may fail to focus on the important information.

Providing too little information will lead to incomplete or even wrong answers. Thus, in good AI application design, one focuses on selecting the relevant information rather than trying to cut down the amount of tokens as much as possible.

This is particularly critical in retrieval-augmented generation, document analysis, enterprise assistants, and in many cases where the AI application is interacting with a long history of conversation.

What Factors Affect Token Count?

The amount of words within a document does not determine the amount of tokens. Tokenization is language dependent and furthermore is affected by the structure of a text (e.g. long sentences), special characters and codes (like programming code) as well as by the used tokenizer.

Language and Text Length

The word count does not always determine the number of tokens in content. Long content generally has more tokens, but the number of tokens relative to the number of words can vary greatly between languages. This is because tokenizers are created for specific vocabularies and are trained on specific patterns in language.

Even within the same language, word length (especially long uncommon words) can cause unusual token distributions between regular text and technical documents.

Code, Numbers, and Special Characters

Code and structured information like this can be very different from normal text when it comes to tokenization. For example, variable names and their surrounding operators and brackets can all count as one token.

When looking at code for tokenization purposes, comments do not contribute to the overall token count as much as they would to the line count.

Premarked data such as tables, JSON, URLs, identifiers, etc. are typically much longer than a similar amount of prose data.

Different Tokenizers and AI Models

There is no single universal tokenizer. A wide array of tokenizers are currently implemented in AI models and model families, each using their own vocabulary to produce unique results.

Developers can only rely on their estimation of token usage if they use the exact same Tokenizer for estimation and for processing of input text by the AI model.

This also needs to be taken into account when comparing the limits of models and the costs and usage limits of API services. A token limit is always only valid in the context of a specific model / tokenizer.