What is a Context Window, and why does it matter?

Ray Poynter, 20 May 2025

One key variable that differentiates one large-language model (LLM) from another is the size of its context window. In this post, I explain what a context window is and why it matters.

What is a context window?
The context window is the active memory for a conversation with a chatbot. Every prompt you type and reply you receive are stored in that window. If the conversation outgrows the window, the earliest turns are “pushed out,” and the model begins to forget them.

When you paste text for an LLM to analyse, that text also has to fit in the window. A larger context window, therefore, allows:

• Longer, more coherent conversations, and
• Bigger chunks of source material to be processed at once.

Rapid growth in size
When ChatGPT was released to the public in November 2022, it offered a 4,096-token context window. Today, windows of 100,000 tokens or more are common. Claude 3, for example, has 200,000 tokens, while the largest of Google’s Gemini models has up to two million tokens. (LLMs count in tokens rather than words; 1 word is approximately 1.3 tokens.) Different models from the same company have different context window sizes. For example, OpenAI ChatGPT GPT-4o has a context window of 128,000 tokens, GPT-3o has a window of 200,000 tokens, and the current flagship has a one-million-token context window.

What happens when you upload a file?
If the file is small enough to fit, it lives entirely inside the context window and can be queried directly.

Larger files are usually chunked and placed in a vector store with a search index. When answering your prompt, the LLM retrieves only the most relevant chunks into the window. That means if you upload, say, the transcripts of many focus groups, you may need to:

1. Explicitly ask the model to summarise each group first, then
2. Work with those summaries (which do fit in the window) for the deeper analysis.

A rule-of-thumb example
I find that an 80-minute focus group typically produces about 9,000 tokens of transcript. A 100,000-token window can therefore hold roughly ten such groups at once. With a two-million-token window, you are unlikely to hit the limit in normal qualitative projects.

Why it matters

• Fewer workarounds – less need to split documents manually.
• Greater consistency – the model can keep more of the earlier conversation “in mind”, reducing contradictory answers.
• Richer analyses – long documents (e.g. multiple reports or large sets of transcripts) can be ingested wholesale.

In short, context-window size sets the ceiling on how much context the AI can actively process, so it is worth checking before you choose a model for your next project.

Related