The Context Window Is a Budget. Here Is How to Spend It

The agent worked. On a short task it read the request, called two tools, wrote a sensible answer. Then somebody gave it a long one, and by the ninth step it was repeating a tool call it had already made, ignoring a constraint stated in the first message and confidently contradicting something it said four turns earlier.

Nothing broke. The prompt is the same prompt, the model is the same model, and the code path is identical. What changed is the shape of what the model can see, and the shape got worse in a way that no error will ever tell you about.

The useful frame for this, and the reason the term has stuck, is that the context window is a budget rather than a container. A container is either full or it is not. A budget forces the question that actually matters: what did that token buy, and what did it displace?

What is context engineering, and how is it different from prompt engineering?

Prompt engineering asks what to say. Context engineering asks what the model should be able to see when you say it, which turns out to be a much larger question with a much larger surface for getting it wrong.

The context engineering guide at Prompting Guide defines it as designing and optimising the instructions and relevant context for a model to do its task, and lists what that context is made of: system prompts, user input with structured delimiters, dynamic elements like the current date, structured input and output schemas, tool definitions, few shot demonstrations, retrieval results, short term state and long term memory.

That list is the whole subject in one line. Every item on it consumes the same finite budget, and only one of them is what most people mean by the prompt.

The formal treatment arrived shortly after the practice did. A Survey of Context Engineering for Large Language Models, which reviews more than 1,400 papers, splits the field into foundational components, meaning retrieval and generation, processing and management, and then the implementations built on top: retrieval augmented generation, memory systems, tool integrated reasoning and multi agent designs. Its most interesting observation is an asymmetry: models understand complex context far better than they produce long structured output from it, which is a good reason to spend budget on what goes in rather than expecting the model to organise its way out on the far side.

Where does the budget actually go?

Here is a worked example for a single turn of a customer support agent. These figures are modelled rather than measured, using the sizes each component typically reaches, and the point is the proportions rather than the absolute numbers. Count your own with a token counting endpoint before making decisions on them.

Two things fall out of that table that surprise people the first time they measure it. The first is that the request itself, the thing everyone spends their afternoon rewording, is under one percent of the budget. The second is that the largest single line is conversation history, it grows without limit by default, and it is almost entirely made of tool outputs whose value expired the moment they were used.

Cost follows the same shape. A system prompt of five thousand tokens is billed on every request, and ten tools at a hundred tokens of schema each is a thousand tokens on every call whether the model uses them or not, as the token counting documentation makes explicit. We went through what that compounds to across a real workload in the piece on why the bill beat the estimate, and the answer was almost always a fixed component nobody was counting.

Why is a bigger context window not a free win?

Because the model does not use all of it equally, and this is measured rather than argued.

The reference result is Lost in the Middle: How Language Models Use Long Contexts, published in TACL in 2023 by Liu and colleagues at Stanford. They took multi document question answering and a key value retrieval task, then moved the document containing the answer to different positions among distractors. Performance was highest when the relevant information sat at the beginning or the end of the context and degraded significantly when the model had to reach into the middle. The finding held even for models explicitly built for long contexts. Their conclusion is worth stating without softening: current language models do not robustly make use of information in long input contexts.

Anthropic's engineering team describes the same territory from the practitioner's side in its note on effective context engineering for agents, and its framing is the one worth adopting. Context is a finite resource with diminishing marginal returns. The model has an attention budget that depletes with every token added, because the transformer creates relationships between every pair of tokens and that relationship count grows as the square of the length. As the token count rises, the ability to recall accurately from the context falls, an effect the post names context rot.

Put those together and the practical rule is uncomfortable for anyone who solved a problem by pasting more in. Adding a document that might help also pushes the document that does help further from an edge, and dilutes the attention available to it. There is a crossover point past which each additional token has negative expected value, and nothing in the interface tells you where it is.

The four moves, and what each one destroys

Every technique in this field is one of four operations on the budget. Naming them this way is useful because it makes the tradeoff explicit: each one buys space by giving something up, and knowing what it gives up tells you when not to use it.

Select: load less in the first place

Retrieval is selection. So is choosing which tools to expose for a given task, and so is trimming a system prompt. The strongest version is what Anthropic calls just in time retrieval: rather than pre loading everything the agent might need, keep lightweight references and let the agent pull the full content through a tool when it turns out to need it.

When it helps: when the relevant subset is small and identifiable, which covers most document grounded work. What it destroys: anything the selector missed is invisible, and the model cannot ask for what it does not know exists. Selection failures are silent and look exactly like model failures.

Compress: keep the meaning, drop the tokens

Summarisation, in its various disguises. The version that matters for agents is what the same Anthropic post calls compaction: as a conversation approaches the window limit, summarise it and restart with the summary in place of the transcript.

When it helps: long running sessions where the early turns matter thematically but not literally. What it destroys: specifics, and always the specifics you did not think to preserve. Identifiers, exact figures and the precise wording of a constraint are the first casualties, which is why a compaction step should be told explicitly what categories to keep verbatim.

Isolate: give the work its own window

Sub agents. A specialised agent handles a focused task in a context window of its own and returns a condensed result, so the detailed work never touches the coordinating context at all. The separation of concerns is real and it is the only move that increases total working capacity rather than rationing it.

When it helps: research, search and any step that generates far more intermediate material than conclusion. What it destroys: shared understanding. The sub agent cannot see what the parent knows unless you pay to send it, and two sub agents cannot see each other at all, which produces contradictory work that nobody notices until integration. This is the same coordination problem we covered in what an orchestration layer has to own.

Persist: move it outside the model

Structured note taking. The agent writes state to a file or a store, and reads it back when needed, so the context window holds a pointer rather than the content. Anthropic describes agents using this to track progress and dependencies across long tasks.

When it helps: anything with a plan, a checklist or accumulating findings. What it destroys: immediacy. Persisted information is only present if something retrieves it, so a note the agent forgets to re read is worse than no note, because the plan says the step is handled.

What does the same task look like budgeted?

Take a twenty step agent task that ends badly: by step fifteen the context holds every tool result from every previous step, the original instructions are 14,000 tokens behind the current position, and the model has started ignoring one of them.

Now apply the four moves without changing the model or the prompt.

That last row is doing more than it looks. Because retrieval from the middle is the weak position, re reading the plan at each step puts the constraints back at the recent end of the context where the model attends to them best. It is not a memory trick. It is positioning, and it works because of the Lost in the Middle result rather than in spite of it.

What should you cut first?

Conversation history, almost always, and specifically the tool outputs inside it. This is the recommendation that meets the most resistance and it survives scrutiny better than the alternatives.

The reason is that tool output has a very short useful life. A search result is needed at the moment the model reads it and then, in the overwhelming majority of cases, never again. A database row is needed to answer one question. Keeping the raw payload for the remaining fifteen steps costs its full token weight on every one of those calls and buys almost nothing, while the conclusion drawn from it, which is what the model actually needs later, costs a fraction and is already in the assistant's own message.

The order after that follows the table above. Retrieved documents next, because a twelve chunk retrieval where two chunks carry the answer is ten chunks of dilution. Then tool definitions, since an agent given ten tools for a task requiring four pays for six schemas on every call and is measurably more likely to pick the wrong one. System instructions last, and if you find yourself cutting there, the real problem is that the instructions are carrying work that belongs in code. We went through what belongs in that slot in what to put in a system prompt and what to leave out.

Which symptom points at which move?

The four moves are not interchangeable, and picking the wrong one is how teams spend a week compressing a window whose problem was selection. The symptoms are distinguishable if you look at the right thing, which is the transcript rather than the output.

The row that catches people is the first. An agent inventing an answer looks like a model quality problem and gets treated with a stronger model, a sterner instruction or a temperature change. None of those help, because the material was not in the window. The transcript tells you immediately: if the retrieval step returned three chunks and none of them mention the thing the answer asserts, you have a selection failure wearing a hallucination costume.

The second row is worth a note on why the drift starts around a specific point rather than gradually. Nothing special happens at turn ten. What happens is that history crosses the point where it dominates the window, and from there every additional turn pushes the instructions further from either edge. The degradation feels sudden because the position effect is not linear, not because the mechanism switched on.

Does this replace prompt engineering?

No, and the framing that it does is the one thing in this field worth pushing back on. LlamaIndex's write up on context engineering and the techniques to consider puts the relationship correctly: the discipline is about curating the most relevant information wherever it comes from, and it names workflow decomposition and structured outputs as the mechanisms, meaning breaking a task into steps each with its own optimised window rather than cramming everything into a single call.

Wording still matters inside every one of those steps. What changed is that wording is now one line item in a budget rather than the whole exercise, which is the same shift we described when agent loops arrived and the prompt stopped being the unit of work. If you are exposing tools to an agent rather than only text, the schemas you write are context too, and every field description in them is paid for on every call. That is worth remembering when designing the tool surface you expose over MCP: a verbose schema is a permanent tax on every request the agent makes, and a terse one that the model misreads is worse.

How would you know it is working?

Measure two numbers before and after any change, and refuse to reason about the second without the first.

The first is tokens per completed task, not tokens per call. Per call figures reward you for splitting work into more calls, which is not the same as doing less work. The second is a task success rate on a fixed set of representative jobs, run before and after. Context changes are exactly the kind that improve the average while breaking one category, because the thing you compressed away mattered to that category alone.

If the success rate holds and the token count falls, you have found real slack. If both fall, you cut something load bearing and the table above will usually tell you which line it came from. And if the token count falls while the success rate rises, which happens more often than people expect, then you were not short of context. You were drowning the model in it.