Understanding context windows: why more is not automatically better
A million tokens sounds like the end of all limits. In practice it is not the size of the window that decides, but what sits inside it.
The context window is a language model's working memory: everything it can see at once in a single pass. The question, the system instruction, attached documents, the conversation so far — all of it counts.
Why large windows still disappoint
Models do not weigh every position in the context equally. Information at the start and end is used more reliably than material in the middle — an effect described in research as “lost in the middle”. Dumping in two hundred pages and hoping for a detail on page 96 works against that property.
What helps instead
- Retrieve first instead of attaching everything: select the relevant passages deliberately.
- Put instructions at the end when a lot of material precedes them.
- Summarise long histories instead of carrying them along in full.
- Provide structure — headings and source labels help the model attribute information.
Then there is the plain economic point: context costs money and time. A prompt that constantly ships 300,000 tokens is several times more expensive and slower than one that sends 8,000 well-chosen ones at the same quality.
The right question
Not “does it fit?” but “does it belong?”. Large windows are a convenience for cases where selection is hard — they do not replace it.
Author
Research, context and fact-checking for everything published on kiprozess.com.
Keep reading
AI governance: what teams should document now
Between compliance theatre and ignoring the topic sits a short list of things every team should know anyway. A pragmatic minimum.
Building a small RAG system: what you actually need
A RAG system sounds like heavy infrastructure. For a small project, four building blocks are enough: documents, an embedding model, a vector database, and a language model. Here's how they fit together conceptually, using Google Gemini and ChromaDB.
Agents in the mid-market: what really happens after the pilot
Almost every company is running an AI pilot by now. Few make it into day-to-day operations. Why that is — and what the exceptions do differently.