RAG isn’t dead. Your RAG architecture from 2024 is.
Every time a lab ships a bigger window, the same take goes around: stuff everything in the prompt and delete your vector database. And every time, people running production systems quietly ignore it, because the argument never survives contact with a billing dashboard.
The windows really did get huge. DeepSeek V4 went GA on July 20, 2026, with V4-Pro at 1.6T total parameters (49B active) built around a native 1M-token context, followed by V4-Flash-0731 eleven days later. Claude Opus 4.6 ships a 1M-token window. GPT-5.6 lists 1.05M. Gemini has been shipping 1M-and-up contexts across its whole lineup, cheapest tier included.
So the capability is real. The question is what it costs per call, and whether the model actually reads the middle of what you sent.
What a Full 1M-Token Call Actually Costs
Let’s do the math instead of hand-waving it. Say you stuff 900K tokens into the window on every request: docs corpus, codebase, some tickets.
- Claude Opus 4.6 at $5 per million input tokens: $4.50 per call
- GPT-5.6 Sol, where requests above 272K input tokens bill at $10/M: $9.00 per call
- DeepSeek V4-Pro at roughly $0.435/M cache-miss input: $0.39 per call
Now multiply by traffic. A modest internal assistant doing 5,000 queries a day on Opus costs $22,500 daily, roughly $675K a month to serve a few hundred employees.
Run the same workload with retrieval pulling 15K relevant tokens instead of 900K and the input bill drops to $0.075 per call, about $375 a day. Same model, same answers, 60x cheaper.
Prompt caching softens this when the corpus is static. Cache hits on DeepSeek V4 cut input costs by one to two orders of magnitude, and OpenAI cached input runs $0.50/M against $5 standard. But caching only pays when the prefix stays identical across calls. The moment context is assembled per user or includes anything fresh, you’re back to full price. More on that tradeoff in my writeup on cutting LLM inference costs without sacrificing quality.
Prices fall roughly 10x a year for equivalent capability, per the trend llm-stats.com tracks. That will keep eroding this argument. It hasn’t erased it yet.
Recall Still Degrades in the Middle
Cost is the boring objection. Quality is the interesting one.
The “lost in the middle” effect that Liu et al. documented in 2023 has been softened by better training, not eliminated. Long-context models tested on RULER still show recall falling as input length grows, and facts sitting at 30-70% positional depth take a measurable hit versus the same fact near the start or end.
Multi-fact retrieval is harder than single-fact. Pulling three related items scattered across 800K tokens and reasoning over them together is a different task from finding one string, and the benchmark gap is wide.
Practical translation: effective context is shorter than the spec sheet. A 1M window is a capacity limit, not a promise of uniform attention. Treating it as a promise is how you ship something that answers correctly in testing (answer at the top of the prompt) and hallucinates in production (answer at token 430,000).
Latency compounds this. Time-to-first-token scales with prefill, so a near-full window puts a visible pause before every answer. Fine for batch analysis. Users notice on anything interactive.
When Each Approach Wins
Here’s how I actually decide, per workload:
| Scenario | Best fit | Why |
|---|---|---|
| One-shot analysis of a 400K-token codebase | Long context | Cross-file reasoning breaks when chunked, and cost is bounded |
| Single large document (contract, spec) | Long context | All of it is relevant; retrieval adds a failure mode |
| Support bot over a 50M-token knowledge base | RAG | Fits no window, and per-query cost matters |
| Data that changes hourly | RAG | Fresh corpus every call kills your cache hit rate |
| Per-user document access rules | RAG | Retrieval filtering is enforceable; prompt filtering is a suggestion |
| High-volume production Q&A | Hybrid | Retrieve generously, let the window absorb imprecision |
| Multi-hop questions across entities | Hybrid + graph | Vector search alone misses relationships |
Access control deserves emphasis. Dump a shared corpus into context and instruct the model to only answer from documents the user may see, and you’ve built a permission system out of vibes. Retrieval-time filtering is the only version that survives an audit.
The Hybrid Pattern That Became Standard
Here’s what won: retrieval is no longer about squeezing into a tiny budget. It’s about filling a large one intelligently.
The 2024 pattern was top-5 chunks at 512 tokens each, precision-obsessed, because 8K windows punished anything extra. The 2026 pattern is top-50 to top-200 results at 1K-2K tokens each, landing between 50K and 200K tokens of context. Optimize for recall, then let the model do the last-mile filtering a reranker used to do badly.
That changes downstream choices. Chunks get bigger and keep more surrounding context, covered in chunking strategies for RAG. Casting a wider net makes production hybrid search matter more, not less. Ordering inside the assembled prompt becomes a real design decision, which is the core of context engineering. For questions hopping across entities, graph RAG fills gaps pure vector similarity never closes.
Rerankers didn’t disappear. Their job shifted to ordering and deduping a big candidate set rather than brutally cutting it to five.
What I’d Build Today
Start with long context when the corpus is small, static, and fits. Skip the vector database, ship faster, accept the low-volume cost.
Add retrieval the moment any of these hit: the corpus outgrows the window, traffic makes per-call cost hurt, data changes faster than you rebuild a cache, or different users see different things.
Then measure effective context on your own data. Place a known fact at 20%, 50%, and 80% depth in a realistic prompt and check whether the model finds it. That five-minute test has changed more of my decisions than any leaderboard.
The line between “just use context” and “you need retrieval” moved a long way in two years. It moved. It didn’t disappear.
Frequently Asked Questions
Is RAG obsolete now that models have 1M-token context windows?
No. Large windows removed the pressure to be surgically precise, but they didn’t solve freshness, per-query cost at scale, access control, or corpora larger than the window. RAG now feeds a big context generously instead of squeezing into a small one.
How much does it cost to fill a 1M-token context window?
Depends heavily on the model. At current rates, a 900K-token input runs roughly $4.50 on Claude Opus 4.6, about $9.00 on GPT-5.6 Sol (which bills long requests at a higher tier), and around $0.39 on DeepSeek V4-Pro. Multiply by your daily query volume before committing to the pattern.
Does lost-in-the-middle still affect 2026 models?
Yes, though milder than in 2023. Benchmarks like RULER consistently show recall dropping as input grows, with the steepest losses for facts in the middle of the window and for questions needing several scattered facts at once.
When should I use long context instead of RAG?
Use it for one-shot analysis over a single coherent body of text: a codebase you’re auditing, a long contract, a research paper set. Low query volume, static content, and reasoning that spans the whole document favor stuffing the window.
What does a modern hybrid RAG pipeline look like?
Retrieve broadly with hybrid keyword plus vector search, pull 50-200 candidates instead of 5, rerank for ordering rather than aggressive cutting, and assemble 50K-200K tokens with the strongest material at the start and end. Filter by permissions during retrieval, never in the instructions.