What is a context window? Why your AI chat forgets the start

A computer chip, standing for the AI's limited working space

You spend twenty minutes with a chatbot planning a family trip to Penang. Early on you told it two things that matter: no seafood for your youngest, and you are driving, not flying. An hour of back and forth later, it cheerfully suggests a hawker centre famous for chilli crab and a cheap flight out of Changi.

It did not ignore you on purpose. It ran into the edge of something called the context window, and once you can picture that edge, a lot of strange chatbot behaviour stops being strange.

The desk, not the library

A language model has two very different kinds of “knowing”.

The first is everything it absorbed in training: an enormous amount of text, read long before you opened the app. Think of that as a huge library the model studied years ago. It cannot look anything up in there now. What it learned is baked in.

The second is the context window. Anthropic’s documentation for its Claude models describes it as “all the text a language model can reference when generating a response, including the response itself”, and calls it a “working memory”, separate from the training data. That is the desk. Your messages, the AI’s replies, any document you pasted, any hidden instructions the app adds: all of it has to fit on the desk for the model to use it in this answer.

The desk is measured in tokens, the small chunks of text a model reads (we explained those in what is a token in AI). And the desk has a fixed size.

[[photo:1]]

How big is the desk?

Much bigger than it used to be, and Singapore’s own model shows how fast that changed.

AI Singapore built SEA-LION, a family of language models trained for Southeast Asian languages. Its first generation had a context length of 2,048 tokens, according to the project’s own repository. A recent release, Qwen-SEA-LION-v4.5-27B-IT, lists a context length of 262k tokens on its model card. That is more than a hundred times more desk between the first generation and this one. The big commercial models have gone further: Anthropic lists several current Claude models at a 1M-token context window.

So why did the Penang planner still forget about the seafood?

Two ways a desk lets you down

Things fall off the edge. Every turn of a conversation goes back on the desk. Anthropic’s documentation puts it plainly: each user message and each reply accumulates in the window, and each new reply is generated from all the previous history plus your latest message. A long enough chat fills it. What happens next depends on the app. Anthropic notes that chat interfaces such as claude.ai “can also manage the context window on a rolling ‘first in, first out’ basis”, which means the oldest material is the first to go. Your seafood rule was in the opening message. It was the first thing pushed off the desk.

Things get buried. This is the less obvious one, and it matters even when nothing falls off. A bigger desk is not a tidier desk. In a study first posted in 2023 and later published in the journal Transactions of the Association for Computational Linguistics, researchers found that models did best when the relevant information sat at the beginning or end of the input, and that performance “significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models.”

A 2025 technical report from the database company Chroma tested 18 models and gave the effect a memorable name, context rot. Its finding was that models “do not use their context uniformly” and grow “increasingly unreliable as input length grows.” Anthropic’s own documentation now uses the same term and says “more context isn’t automatically better.”

Why would a machine get worse with more to read? Anthropic’s engineering team offers a plain reason: in the transformer design behind these models, every token can attend to every other token, which creates “n² pairwise relationships for n tokens.” Double the text and the number of relationships the model has to juggle roughly quadruples. They describe the model as having an “attention budget” that every new token draws down.

That last phrase is the one to keep. The desk is big. The attention is not.

What this changes about how you use AI

Once you think in desks rather than memories, three habits follow, and none of them needs a paid plan.

  • Start a fresh chat for a new job. The trip plan and the cover letter do not belong on the same desk. A new conversation clears the clutter, so the model’s attention goes to what you are doing now.
  • Put the rules where the model looks hardest. The research points to the start and the end. State your constraints in the opening message, and if a chat runs long, repeat the important ones at the bottom of your latest message: “Reminder: no seafood, we are driving.”
  • Paste the part, not the pile. Asking about one clause in a tenancy agreement or one section of a school handbook? Paste that section, not all forty pages. Anthropic’s engineers frame good practice as finding “the smallest possible set of high-signal tokens” for the task. That is advice for developers building agents, but it works just as well for someone at a laptop on a Sunday night.

A useful trick when a long chat starts to wobble: ask the AI to summarise the decisions and rules so far in a short list, check it yourself, then paste that list into a fresh chat. You have just cleared the desk and kept only what matters.

The next time a chatbot forgets something you told it, try not to think “it is stupid” or “it is ignoring me”. Think: that was on the desk an hour ago, and the desk has moved on. Then put it back.

Was this useful?

Sources

  1. Context windows · Anthropic (Claude Platform Docs)
  2. Effective context engineering for AI agents · Anthropic
  3. Lost in the Middle: How Language Models Use Long Contexts · Transactions of the Association for Computational Linguistics (Liu et al.)
  4. Context Rot: How Increasing Input Tokens Impacts LLM Performance · Chroma
  5. Qwen-SEA-LION-v4.5-27B-IT model card · AI Singapore
  6. SEA-LION: Southeast Asian Languages In One Network · AI Singapore