Every large language model has a context window: a maximum number of tokens it can process in a single request, counting both the input you provide and the output it generates. Many models also cap output length separately, usually well below the full context window. Understanding how these limits affect real performance, not just whether a request technically fits, matters for anything that depends on a model handling substantial context reliably.
What actually counts against the limit
The token limit covers the entire request: your system prompt, any conversation history, any retrieved documents provided as context, and the model's own generated response, all combined. That's why, as covered in why AI agent context keeps growing, a long conversation or a request with substantial retrieved content can consume a meaningful share of the available limit before the model even starts generating its actual response.
Why hitting the hard limit isn't the only performance concern
A request that technically fits within the token limit isn't guaranteed to perform as well as a shorter one. Models can show a pattern called "lost in the middle," where information buried in the middle of a very long context is used less reliably than information near the beginning or end. A technically valid but very long context can therefore produce worse results than a shorter, more focused one, even without exceeding any hard limit.
Why this matters for how you actually build with these systems
Retrieval quality matters more than retrieval volume
Given both the token limit and the lost-in-the-middle effect, retrieving fewer, more precisely relevant pieces of context tends to outperform retrieving as much as technically fits. We cover this directly in why RAG gives wrong answers even with the right documents.
Conversation history needs active management
A long-running conversation will eventually approach the token limit if the full history is kept indefinitely. That's why summarizing or trimming older turns is a necessary practice for any system expecting extended interactions, not an edge case to handle later.
Cost scales with tokens, not just with requests
Both input and output tokens generally count toward cost, so a system that sends more context than needed, or generates longer responses than necessary, pays for the excess. This connects directly to the cost discipline covered in context growth and cost.
Different models have genuinely different limits
Token limits vary meaningfully across models and providers. For use cases that need to handle substantial context, that's a real factor in model selection, alongside the other criteria covered in how to choose between Claude, GPT, and Gemini. A larger stated limit is also worth testing for actual quality at that scale, rather than assuming the model performs equally well across its full range.
How we approach this
We design context management (retrieval precision and conversation summarization) around both the hard token limit and the practical quality effects of context length. We don't assume a request that technically fits will perform as well as a shorter, more focused one.