The Race for Longer AI Context Windows

A context window is the amount of text an AI model can hold in mind at once: the conversation so far, the documents you paste in, the code it is working on. A few years ago, a window of a few thousand tokens was normal. Today the leading models handle a million or more, and the race is on to go further. But longer windows are not just a software feature. They put heavy pressure on memory, chips, and cost. For more on the hardware side of AI, see our AI infrastructure and data center coverage.

Where the race stands

By mid-2026, a window of around 1 million tokens has become standard at the top end. One 2026 explainer says flagship models from OpenAI, Google, Anthropic, DeepSeek, and Alibaba all offer around 1 million tokens, roughly 750,000 words or 1,500 pages, enough for a large codebase, long videos, or a stack of contracts in a single prompt. The same source notes that Meta’s open-weight Llama 4 Scout advertises the largest window, at 10 million tokens.

Because most top models have converged on the same headline number, the explainer argues the practical differences now come down to reasoning quality, multimodality, and price instead of raw window size. Specific model rankings change quickly and sources disagree on details, so check each vendor’s documentation before quoting a particular limit.

Why longer windows matter

Bigger windows change what AI can do. Instead of splitting a task into chunks, a model can read an entire code repository, a long legal filing, or months of work history in one pass. That fits the shift toward agents we described in why AI inference is becoming the next big infrastructure challenge: agents accumulate context as they plan, call tools, and revise, so each step benefits from remembering the last.

It also changes product design. A long window can reduce the need to build retrieval systems for some tasks, and it lets users work with whole documents instead of excerpts.

The hard part: attention is expensive

The reason windows were short for so long is the way transformers work. Standard self-attention compares every token with every other token, so cost grows quickly with length. One technical explainer puts it simply: doubling the context roughly quadruples the compute. Optimizations such as FlashAttention, sparse attention, and ring attention reduce the constant factors, as another reference notes, but do not change the quadratic growth rate of full attention.

Researchers have measured the pain directly. In a paper on retrieval-based attention, the authors report that with a 1-million-token prompt on Llama-3-8B, generating every token without a cache takes 1,765 seconds, with over 96 percent of the time spent on attention. That is why practical systems rely on caching, and why caching creates a second problem.

The memory wall

The cache that makes long-context generation possible is the KV cache, which stores the model’s working memory of every token so far. The same paper finds that a 1-million-token context needs about 125 GB just to store the KV cache, far beyond a high-end A100’s 80 GB. Another explainer says that for a 10-million-token sequence the cache would run into terabytes.

This connects directly to the hardware shortages we covered in why semiconductor supply chains matter for AI. Long context pulls on high-bandwidth memory, DRAM, and storage at the same time, which is why chipmakers are changing designs. Google’s eighth-generation TPU 8i triples on-chip memory, and NVIDIA’s BlueField-4 platform targets context memory storage for long-running agents, as described in why NVIDIA has become more than a GPU company. TrendForce adds that when GPU memory is insufficient, systems must discard and recompute the cache, which raises latency and cost.

How engineers are stretching the limit

Several techniques are making long windows practical.

Distributing the sequence. Ring attention spreads a long sequence across many GPUs arranged in a ring. A 2026 overview calls it arguably the key enabler of the 1M+ token era, because no single GPU has to hold the whole cache.

Shrinking the cache. Methods like grouped-query attention share keys and values across heads. One guide says an 8-to-1 ratio shrinks the KV cache eightfold. A recent paper on sparse attention notes that DeepSeek-V4 cut per-token cache by roughly 10 times compared with its predecessor, though capacity is still a bottleneck as contexts grow.

Caching repeated prefixes. Providers can store the processed cache for a static prompt and reuse it. A reference on context windows says prompt caching can cut costs by up to 90 percent on the cached portion, which matters for agents with long system prompts.

Hybrid designs. Some architectures mix attention with other layers to cut memory. One example cited is a hybrid model that reduces its cache from 32 GB to 4 GB at 256K tokens, though that figure comes from a secondary blog.

A bigger window is not always a better memory

Advertised size and usable size are different things. A 2026 ranking that cites the NVIDIA RULER benchmark says models reliably use only 50 to 65 percent of their advertised window. A separate analysis of long-context benchmarks claims the gap between the best and worst 1M-capable models is about 60 points on one test. These are secondary sources and the figures depend on the benchmark, but they agree on the main point: accepting a million tokens does not mean using all of them well.

One familiar failure is the lost-in-the-middle effect, where information buried in the middle of a long prompt is recalled less reliably. For builders, this means testing a model on your own data rather than trusting the headline number.

Cost and the RAG question

Long context is also expensive. A technical guide estimates that processing 1 million tokens costs roughly $2 to $10 per query with frontier models, while retrieval-augmented generation (RAG) sends only the relevant chunks at a fraction of the cost. Another source argues the economical pattern is to retrieve the relevant few thousand tokens and reserve the giant window for the rare cases that truly need it.

The practical answer is a mix. Use the full window for global reasoning over a whole document or codebase, and use retrieval for repeated factual lookups. This mirrors a split Apple describes in how Apple is bringing more AI processing to its devices, where on-device models handle limited context quickly and server models take the longer, heavier work.

What it means for infrastructure

The context race ties into every theme we have followed. Longer windows mean more memory per user, which raises the cost of serving each request. That feeds the capital spending we described in why Big Tech is spending billions on AI infrastructure and the power demand covered in why AI data centers need much more power. A model that supports ten times the context may need far more than ten times the supporting hardware, depending on how well the techniques above work.

The risks and open questions

Much of the evidence on effective context comes from vendor benchmarks and third-party blogs, and results vary by task. Longer prompts also bring security and quality concerns: more text means more room for irrelevant or conflicting information. And it’s not clear that bigger always wins. If retrieval, memory systems, and caching improve, the most useful systems may not be the ones with the largest windows.

The bigger picture

The race for longer context windows is really a race to solve memory. Models can already accept a million tokens or more, but making that practical depends on chips with more and faster memory, smarter software to shrink and reuse the cache, and honest measurement of what models can actually recall. The winners will be the systems that use long context efficiently, not just those that advertise the biggest number. Follow our AI hardware coverage as memory and context keep shaping the next generation of AI infrastructure.