A context window is the maximum amount of text (measured in tokens) that an AI language model can process and attend to in a single interaction. Everything the model "sees" in one exchange — your prompt, any documents you've pasted in, the conversation history, and the model's own prior responses — must fit within this limit. Material outside the context window is invisible to the model. In 2020, context windows were 2,000-4,000 tokens; in 2024-2025, frontier models support 128,000 to 2,000,000 tokens.
The context window is the AI's working memory — large enough to hold a lot, but not infinite.
Where the Concept Comes From
The context window limitation is inherent to the Transformer architecture (Vaswani et al., 2017) that underlies modern LLMs. The attention mechanism — the core of the Transformer — computes relationships between all token pairs in the input. This computation grows quadratically with sequence length: doubling the context length roughly quadruples the compute required.
Early GPT models had 2,048-token context windows (roughly 1,500 words). GPT-3 had 4,096 tokens. GPT-3.5 (ChatGPT's original model) had 4,096 tokens. These limits were practical constraints driven by compute costs.
The context window race began in earnest in 2023:
- Anthropic's Claude extended to 100,000 tokens (roughly 75,000 words)
- Google's Gemini 1.5 Pro reached 1,000,000 tokens (roughly 750,000 words) and later 2,000,000
- OpenAI's GPT-4 Turbo reached 128,000 tokens
New architectural research (Flash Attention, ring attention, sparse attention) has reduced the quadratic scaling problem, making large context windows more computationally feasible.
What Is a Token?
Tokens are the units that LLMs use to measure text. A token is roughly:
- 0.75 English words (so 100 tokens ≈ 75 words)
- A word or sub-word unit in English
- A character in some languages (Chinese, Japanese)
Rough conversions:
- 1,000 tokens ≈ 750 words ≈ 3 pages of standard text
- 10,000 tokens ≈ 7,500 words ≈ 30 pages
- 100,000 tokens ≈ 75,000 words ≈ 1 long novel
- 1,000,000 tokens ≈ 750,000 words ≈ 10+ long books
Most model providers offer token counting APIs or tools so you can measure how much of your context window any given input uses.
Why the Context Window Matters
For users:
The context window determines what the AI can "see" during your conversation. If you're having a long conversation and the conversation history exceeds the context window, earlier parts of the conversation become invisible — the model may give responses that seem to "forget" earlier context because that context has been truncated.
When you paste in a document, it uses context window space. A 20-page PDF might use 10,000-15,000 tokens — leaving less room for conversation.
For developers:
The context window constrains application design:
- How much document content can you include in a single API call?
- How long can a multi-turn conversation go before truncation?
- How many few-shot examples can you fit in a system prompt?
Applications that need to process very long documents (50+ pages) must either chunk them (process sections separately with RAG) or use a model with a large enough context window to handle the full document at once.
A Worked Example
A developer is building a code review assistant. Users paste in a pull request for review.
Small context window (4,096 tokens ≈ 3,000 words):
A large PR with 500 lines of code changes + docstrings might be 2,500-3,000 words. Adding a system prompt (200 tokens) and conversation history from earlier review comments (500 tokens) pushes toward or past the limit. The model truncates the PR, reviews an incomplete view.
Large context window (128,000 tokens):
The same PR is well within the context window. The developer can also include: relevant documentation (5,000 tokens), prior code review standards (2,000 tokens), and examples of good and bad review comments (3,000 tokens). The model has full context to produce high-quality, standards-aligned code review.
Very large context window (1M tokens):
The entire codebase for a small project could fit. The model can reason about how changes in one file affect others without requiring the developer to manually identify relevant files.
Context Window Comparison (2024-2025)
| Model | Context window | Approx. equivalent |
|---|
| GPT-3.5 (legacy) | 16,385 tokens | ~12,000 words |
| GPT-4o | 128,000 tokens | ~96,000 words |
| Claude 3.5 Sonnet | 200,000 tokens | ~150,000 words |
| Gemini 1.5 Pro | 1,000,000 tokens | ~750,000 words |
| Gemini 1.5 Flash | 1,000,000 tokens | ~750,000 words |
| Llama 3.1 (70B) | 128,000 tokens | ~96,000 words |
Costs also vary: larger context windows typically cost more per API call because the attention computation is more expensive.
The "Lost in the Middle" Problem
More context isn't always better. Research by Liu et al. (2023, "Lost in the Middle: How Language Models Use Long Contexts") found that LLM performance on retrieval tasks from long contexts degrades when the relevant information is in the middle of the context — models attend better to information at the beginning and end.
Practical implications:
- For RAG applications, putting the most important retrieved chunks at the beginning or end of the context (not sandwiched in the middle) improves performance.
- Using a very large context window to include everything doesn't always outperform targeted retrieval that puts the most relevant content first.
- For long conversations, key instructions in the system prompt (at the beginning) persist better than key information mentioned once in the middle of a long conversation.
Context Window Management Strategies
For developers:
Chunking and RAG: For documents larger than your context window, split into chunks and retrieve only the most relevant chunks for each query rather than loading the entire document.
Conversation truncation strategies: For multi-turn conversations, implement sliding window (keep the most recent N messages), summarization (summarize older messages and keep the summary), or selective retention (keep all system messages + the last N turns).
System prompt efficiency: The system prompt occupies context window space in every API call. Keep it as concise as possible while maintaining necessary instructions.
For users:
Start new conversations for new topics: Long conversations exhaust the context window. When switching to a substantially new topic, starting a fresh conversation preserves more of the context window for new content.
Summarize before pasting: When pasting in documents, summarize or excerpt the most relevant sections rather than pasting the full text.
Watch for "forgetting": If a model seems to contradict something said earlier in a long conversation, the earlier context may have been truncated. Repeat the key information in the current message.
Context Window vs. Training Data
A common confusion: the context window and training data are different things.
| Feature | Context window | Training data |
|---|
| What it is | What the model can see in one interaction | What the model learned from |
| When it matters | At inference (when you use the model) | At training (before you use the model) |
| Your content in it | Yes, if you paste it in | No (unless you fine-tune) |
| Limit | Hard limit in tokens | No limit — grew until training ended |
| Private data safe? | Goes to the API provider for that session | Would be retained if used for training |
You can include your private data in the context window (as text in your prompt) without it being incorporated into the model's weights — the model reads it for this interaction only, doesn't "learn" it permanently.
Common Misconceptions About Context Windows
"More context window always means better responses."
The "lost in the middle" effect means that very large contexts can dilute attention on the most relevant information. Targeted retrieval (RAG) sometimes outperforms stuffing a large context window with everything potentially relevant.
"The model remembers our conversation next time."
Memory across sessions is a separate feature from context window, implemented by developers. By default, each new conversation starts with an empty context window — the model has no memory of previous conversations unless the developer has implemented session persistence.
"Context window = training data limit."
These are separate. The context window is what the model sees per interaction; training data is what the model was trained on. A model trained on 15 trillion tokens still has a context window of a few hundred thousand tokens per interaction.
Related Concepts
Token: The unit by which context window size is measured — roughly 0.75 English words.
RAG (Retrieval-Augmented Generation): A technique for working around context window limits by retrieving only the most relevant documents for each query rather than including entire knowledge bases.
Attention mechanism: The Transformer component whose quadratic scaling with sequence length is the reason context windows were small — and whose optimization is the reason they're growing.
LLM (Large Language Model): The system with a context window — all modern LLMs have context window constraints.
Frequently Asked Questions
What happens when you exceed the context window?
Depends on the implementation. API calls that exceed the context limit return an error. Chat interfaces typically truncate (silently drop) the oldest messages to stay within the limit, which means the model "forgets" older conversation context. Some interfaces summarize older content before truncation.
Is a 1M token context window really useful?
For the right use cases — analyzing an entire codebase, reading all of a company's documentation, reviewing a 500-page report — yes, dramatically. For typical single-document or single-question use cases, a 128K context window is more than sufficient.
Does everything in the context window get equal attention?
No — the "lost in the middle" research suggests that information at the beginning and end of the context receives more attention. This affects how you structure prompts for long-context applications.
Key Takeaways
- Context window is the maximum amount of text an LLM can process in a single interaction — measured in tokens.
- Tokens ≈ 0.75 English words; 1,000 tokens ≈ 3 pages of text.
- Context windows have grown rapidly: from 2,048 tokens (2020) to 2,000,000 tokens (Gemini 1.5, 2024).
- "Lost in the middle": LLMs attend better to context at the beginning and end — relevant information sandwiched in the middle may be underweighted.
- Different from training data: what's in the context window is what the model sees now; training data is what the model learned from.
- Management strategies: RAG for large documents, conversation summarization for long chats, concise system prompts.
Conclusion
The context window is the most practical architectural constraint affecting how you use LLMs in real applications. Understanding it clarifies why AI assistants "forget" context in long conversations, why pasting 200 pages of documentation rarely works well, and why RAG-based retrieval often outperforms pure context stuffing even when the context window is large. For developers building LLM applications, context window management is a core design concern. For users, understanding the limit helps explain unexpected model behavior and guides better prompting habits.
Try WebSnips free — rather than pasting entire documents into an AI's context window, save and annotate the specific passages that matter: building a curated knowledge base that gives you targeted, high-signal context for your AI interactions.