In this aiengineeringsimplified newsletter post, we will discuss when to use RAG versus a long context window, and why that decision comes down to cost and accuracy math, not whichever approach is trendier this quarter.

A teammate pinged me last week with a one-line proposal: rip out the RAG pipeline. Just stuff the whole knowledge base into the prompt.

Sonnet's context window is big now, so why keep maintaining chunking, a vector store, and a reranker for something the model can apparently just read in one go?

I almost said yes. Then I actually ran the numbers, and the math changed my answer completely.

New here? I'm Divy, and I write practical, engineering-focused content on AI, LLMs, RAG, agents, and ML infrastructure.

Follow along:

Every issue focuses on explaining one technical concept clearly so you can build faster and stay current with the latest developments in AI.

Now let’s get into today’s topic.

Context windows genuinely got bigger. <cite index="12-1">Claude Sonnet sits at 1 million tokens, Gemini 3 Pro at 2 million, and Llama 4 Scout claims 10 million</cite>. That's a real shift from two years ago, and it makes a genuine case: no chunking, no vector database, no retrieval pipeline to maintain.

But "the model can technically read it all" and "this is the right architecture for production" are two different claims, and conflating them is where the expensive mistakes start.

Take a 5MB internal knowledge base, roughly 1 million tokens. <cite index="15-1">Stuff the whole thing into every query and you pay for reading the entire archive on even the simplest request</cite>.

RAG works differently. <cite index="15-1">It only pays for the tokens that are actually relevant to the question</cite>, typically a few thousand instead of a million.

At current Claude Sonnet 5 pricing ($2 per million input tokens through the August 2026 introductory window), a full-context query on that knowledge base costs roughly $2. A RAG query pulling the same 3,000 relevant tokens costs about $0.006. That's the 333x gap. Multiply either number by real production query volume and the architecture choice stops being philosophical.

Here's the part that surprises most engineers: a bigger window doesn't mean the model reads everything equally well.

Research from Stanford and UC Berkeley identified what's now called the "lost in the middle" problem. <cite index="26-1">Model performance on multi-document tasks follows a U-shaped curve: high recall when the answer sits at the very start or end of the context, and a sharp drop when it's buried in the middle</cite>.

This isn't old news that newer models fixed. <cite index="20-1">As of 2026, no production model has fully eliminated the position bias. It's structural to how transformer attention works</cite>. Independent testing on 2026-era long-context models keeps reproducing the same U-shape, and <cite index="17-1">teams that treat a 1-million-token window as a general-purpose RAG replacement have reported fact miss rates as high as 40 percent and latencies stretching to 45 seconds</cite>.

Bigger context gave you more room. It also gave you more room to lose things in.

You don't always need a new pipeline to fix this. One documented mitigation is almost embarrassingly simple: <cite index="23-1">priming the model by explicitly stating that you're about to show it the most relevant sentence in the context</cite>.

On a 200K-token needle-in-a-haystack test, that single prompt-level change <cite index="23-1">lifted accuracy from 27% to 98%</cite>. No new reranker. No new retrieval layer. One sentence of structure.

That's the pattern worth remembering: the fix for a context problem is sometimes architectural, and sometimes it's just telling the model where to look.

Use RAG when:

Use long context when:

Use both when:

Is RAG dead in 2026 now that context windows are so much bigger? No. RAG is no longer the automatic default for every architecture, but it remains the better choice whenever query cost and traceability matter, and whenever the knowledge base is far larger than any single query needs.

Does a bigger context window fix the lost in the middle problem? No. The position bias is structural to transformer attention, not a capacity limit. A 1-million-token window has more middle to lose facts in, not less of a bias.

What's the cheapest way to avoid the lost in the middle problem? Reorder retrieved content so the most relevant material sits at the very start or end of the prompt, and explicitly prime the model to look for the most relevant sentence before it answers.

How much cheaper is RAG than full-context stuffing? On a roughly 1-million-token knowledge base, retrieving only the relevant chunks instead of sending the whole archive can cost around 300 times less per query, depending on how narrow the retrieval is.

Building something that has to choose between RAG and long context? Reply and tell me what you're weighing — I'll dig into specific tradeoffs in a future issue.