A conversation that starts at a couple thousand tokens can be paying for many times that by the tenth exchange. That isn't a bug; it's how conversational AI systems work by default. Every turn typically resends the full conversation history, plus any newly retrieved documents, plus the growing record of what the agent has already tried. Left unmanaged, that growth is silent until the bill or the latency makes it obvious.
Why context grows faster than it looks like it should
Each new turn in a conversation doesn't just add its own content; it re-sends everything that came before, because the model needs the full history to respond coherently. (Prompt caching, offered by several providers, can reduce the cost of re-sending an unchanged prefix, but it doesn't remove the growth itself.) Add retrieved documents from RAG, tool call results, and an agent's own intermediate reasoning steps, and a session's context window usage compounds turn over turn in a way that's easy to underestimate when you're only testing short conversations.
The costs this actually creates, beyond the bill
- Rising per-session cost that isn't visible in aggregate monthly spend until it's already a pattern. This is exactly what per-request cost monitoring is meant to catch early.
- Slower responses, because processing a larger context takes longer, however efficient the rest of the system is.
- Degraded answer quality. Models can underweight information buried in the middle of a very large context, so a longer context isn't strictly better even when cost and latency aren't a concern.
What actually keeps context growth under control
Summarize older turns instead of keeping them verbatim
Once a conversation exceeds a reasonable length, older turns can be compressed into a summary that preserves the information the model still needs without carrying the full original text forward indefinitely. This trades a small amount of fidelity for a meaningful reduction in ongoing cost.
Be deliberate about what actually needs to persist
Not every tool result or intermediate reasoning step needs to stay in context for the rest of the session. Being explicit about what's actually still relevant, rather than defaulting to "keep everything, just in case," is one of the highest-leverage places to cut unnecessary growth.
Retrieve narrowly, not broadly
The instinct to retrieve more context "to be safe" works against you twice: it costs more, and it can dilute the model's attention on the part that actually matters. We cover this in more depth in why RAG gives wrong answers even with the right documents.
Set explicit limits and monitor against them
A maximum context budget per session, with monitoring against it, turns an invisible cost trend into a visible, actionable metric. Otherwise you discover the problem only when it shows up as a surprising invoice or a complaint about slow responses.
How we approach this
We build context management (summarization of older turns, narrow retrieval, explicit session budgets) into production AI systems from the start, so cost and latency stay predictable across realistically long conversations, not just the short ones that get tested first.