What is a Context Window?

What Is a Context Window?

The term ‘context window’ in AI is used to express the size of a language model’s working memory for information processed within a single interaction.

In addition to the current prompt (the message users are typing), the context for chatbots includes past messages in a conversation. For document analysis and other AI applications that work with documents, the context includes document content, input instructions, retrieved information and other supporting materials (as shown in the figure above).

The context window of a model is of fixed size for a given model. The information supplied to the model can therefore exceed the context window. The application then must cut, truncate, summarize, etc. parts of the information before it can be processed by the AI application. The context window size of a model has implications that go far beyond a model’s specification. The actual working range of an AI application in terms of information size is largely determined by the context window.

Context Windows and Tokenization

The amount of information a model can process is often expressed in terms of tokens (the information that goes into the model) rather than words. As a result, the concept of tokenization is crucial to determining the actual information processing capacity of a model.

The word “token” is referred to in several places in this discussion. A token is a unit of text that is processed by a language model. This may be a word, part of a word, a single character, such as a space, a punctuation character, or even a short text fragment.

For instance, the same written character can be a word in isolation but part of another word in another sentence. The tokenizer can represent that written character as one token in the first case but as part of another token in the second case.

Note that there is no universal ratio of words to tokens, and even for a single language there can be significant variation depending on the model, and even within a single model depending on the input (e.g. technical content vs. chat logs).

How Are Tokens Used in a Context Window?

The model processes information in the available context in the form of relationships between the individual tokens that comprise the input. This includes the prompt, system instructions, prior conversation history, and any other information that comprises the available context.

Note that the user prompt itself will consume tokens in the context window, in addition to system commands, history of prior conversation, retrieved information, and even formatting of prior information that is consumed by the model.

A practical example of this, is retrieval-augmented generation (RAG) which consumes parts of the available context for the retrieved passages, the user’s query and other instructions.

Why Do AI Models Have a Maximum Context Length?

The length of a sequence (context window) is limited by the amount of computation required to process it.

The vast majority of the current state-of-the-art language models are based on the Transformer architecture. These models use self-attention to compute attention weights between different parts of the sequence to determine which tokens are relevant to each other and then compute a weighted sum of these tokens to compute the representation for a given token.

The context window specifies how many tokens can be processed in one batch.

In simple terms, for standard attention mechanisms, the amount of computation required is proportional to the square of the sequence length. Thus, doubling the length of the input sequence to the model requires, approximately, four times as much computation to perform attention.

Note that increasing the context window size is not merely a matter of raising a limit. There are many factors involved including model architecture, memory usage, efficiency during inference, and even hardware.

How Does Transformer Architecture Use Context?

For example, a model can use self-attention to identify that information near the beginning of a piece of text and information near the end of the same piece of text are related to each other.

This can be illustrated with a simple example such as a paragraph containing information near the beginning and the end of the paragraph. The model can use attention to compute the interaction between these pieces of information.

However, longer context is also limited by the way the context is processed by the model in the input sequence. Typically the context window limits how many tokens can be processed in a single attention mechanism call, so longer context means more calls to the same mechanism.

Other information also can be processed in a Transformer architecture. Examples include the pixels in images (in image-generation systems, where the information in attention bands is the information in the pixels).

What Information Takes Up Space in a Context Window?

A common misconception about context is that the user’s input (e.g. message) is the only thing that counts against the context window.

The types of information that will consume context in the context window are the prompt(s) for user input (including system prompts), the previous messages in a conversation, the RAG retrieved information, documents, code, special characters that are used for formatting, etc.

To illustrate this, a user prompt may appear to use a large context window, however there may be additional hidden prompts, previous messages, system prompts, RAG retrieved content, documents, code etc, that are also using the available tokens.

Context Windows and Computing Resources

Increasing the context window size also increases the amount of computational resources required to process long inputs.

For the conventional self-attention (also known as dot-product attention), the number of computation steps grows very quickly with the sequence length. Thus, longer contexts increase the computational resources as well as the latency for the inference.

Longer context also leads to longer response times. In autoregressive generation, each token of the sequence is generated by considering all the tokens that have come before it in the context. Thus, increasing the length of the context also increases the time it takes to process each token in the sequence.

In situations, however, where immediate response is critical, using the entire capacity of the context window may not be sensible for reasons of cost and latency.

How Does Context Length Affect Processing Power?

There is a strong relationship between the length of the context and the amount of computation required to compute attention.

If the input sequence is twice as long as before, i.e. 2n tokens instead of n, then the number of token pairs (i,j) that standard self-attention has to consider will approximately be 4n times as many as before.

While longer context windows impose large computational costs, recent developments in model architectures and efficient inference have made handling of long context windows feasible, sometimes even more efficient than for short context windows of previous language models.

How Does a Larger Context Window Affect Response Speed?

Increasing the length of the context available also increases the time it takes to process a request.

While short prompts are processed quickly, very long contexts result in systems that are computationally very expensive. For applications that require near-real-time processing of AI-generated responses, the length of context can be a very significant consideration.

Thus, developers must balance the maximum length of a context that is available to a model with its processing time, economic feasibility, and the specific requirements of the corresponding application.

Challenges of Long Context Windows

While the longest possible context window might seem the best solution for projects that are required to process lots of information, the increased length of the context poses problems of a different nature. Not all information contained in the context is equally used by the model.

Performance Challenges

Increasing the context length does not necessarily also increase the accuracy with which a model can retrieve information from within a very long context. The model may not be able to make effective use of all the tokens in very long input sequences.

An example of the impact of information organization is in understanding that a model that supports long context windows does not necessarily support good retrieval of information within that window.

Long-context models are even more difficult to test for information retrieval within very long inputs, which led to the creation of a variety of benchmarks that test long-context performance including Needle in a Haystack (NIAH), RULER and LongBench.

Safety and Cybersecurity Challenges

Furthermore, larger context windows increase the attack surface of an application running with LLMs. An AI’s ability to process long context windows increases the potential attack surface of an AI application. By accepting a larger amount of input, a model can become influenced by hidden instructions, including potentially malicious ones. A recent study investigated the impact of increasing the context window on jailbreaking and adversarial attacks of several well-known LLMs.

So context management is also a security concern and retrieved documents, user input, external sources, hidden instructions, etc. should not be automatically treated as trustworthy just because they are included in the model’s context.

Context Window Sizes of Prominent LLMs

As language models have evolved to handle longer pieces of text, the context window size for each model has significantly increased. Often the size for a particular model can change as new versions are released, and sometimes even different versions of the same model family can have different maximum context sizes.

Context window sizes vary significantly across AI models. The following examples are based on specifications published by leading AI providers and industry sources.

The maximum number of input tokens and output tokens, and the model name, are historical specifications from website reference. These are not current specifications for models available in 2026. A model provider can change a model’s input or output limits, as well as update a version number, at any time.

Increases in the context window size of LLMs have become a familiar trend over the last years. Yet the real question for any application of LLMs is whether the extra tokens in working memory actually make a difference for that specific application.

Context Window vs. Memory

A context window is sometimes mistaken for the working memory of AI models, however this is different from persistent memory.

In simple terms, the context window contains all the information the model can use for a given interaction (all the input information the model has in its working memory). Anything outside the scope of the current context cannot be used by the model unless stored and retrieved from memory (persistent memory) by the application.

A model’s memory is not necessarily restricted to the model’s context window. Memory systems for applications can store and even manage information of an application in memory that is even beyond the model’s context window and even retrieve that information later when needed.

In summary, having a large context window does not necessarily mean that a model has persistent memory. It can be the other way around as well: the application can offer memory features even if the underlying model only has a relatively small context window.

Why Does Context Window Size Matter for AI Applications?

Context window size affects the design of AI applications in cases where a large amount of information is processed.

Document analysis: Maximum size of source documents that can be fed to the AI during interaction with it.

RAG systems – it determines how much information you can put into the prompt itself, in addition to the retrieved info and the instructions.

For coding assistants, the size of the context window for the AI model will impact the amount of source code, documentation, errors and project context that can be used for generating code and assisting the developer.

Conversational AI: How much previous dialogue can be stored without being summarized by external memory.

Note that there is a trade-off between maximizing the context for a model’s consideration and limiting it to a certain scope. The latter can be very valuable if it leads to more relevant and higher quality output.

Ultimately, it is a matter of supplying the appropriate information, while also controlling computational resources, latency, relevance, and security.