Cheap Cached Context: When to Skip the Retrieval Pipeline

A policy library of 250,000 tokens now costs about ten Australian cents to put in front of a model on every single call, provided the cache hits. At that price the question stops being "how do we build retrieval for this" and becomes "do we need retrieval at all". For a large class of small-business workflows — contract review, quoting against a product catalogue, answering questions against an operating manual, checking a job file for compliance — the answer as of September 2026 is often no.

That is a genuine change, and it is recent. Here is what moved and where the line now sits.

What changed in the first fortnight of September

Five labs shipped in ten days. The pricing detail that matters for automation work is not the headline input rate, it is the cached-read rate and the context window.

Anthropic launched Claude Fable 5.1 on 1 September 2026 at US$10 per million input tokens and US$50 per million output, with cached-input reads at US$0.25 per million — a reported 75% cut to cache-read pricing. Google followed on 2 September with Gemini 3.8 Flash at US$0.75 input and US$3.75 output, initially priced that way until 31 December 2026. OpenAI's GPT-6 Astra arrived on 3 September with a 1.05-million-token context window at US$10 / US$50. DeepSeek shipped V4.1-Flash on 10 September with a one-million-token window at US$0.15 / US$0.60 and cache-hit pricing reported as low as US$0.003 per million tokens.

Two things happened at once: the window got big enough to hold an entire corpus, and the price of re-reading that corpus on every call fell through the floor.

The arithmetic in Australian dollars

Converted at AUD 1.00 = USD 0.66 and rounded, the September rates look like this:

Now take a realistic SME corpus. A 400-page policy and procedure set, or a few hundred current supplier contracts, or a product catalogue with specifications, runs to roughly 200,000 to 300,000 tokens. Call it 250,000.

On Claude Fable 5.1, that corpus costs AUD 3.79 to read cold and AUD 0.095 to read from cache. At 300 calls a day — a busy internal assistant, or an automation triaging every inbound email against your policies — that is AUD 1,136 a day uncached and about AUD 28 a day cached. Over a month of working days: AUD 25,000 versus AUD 625.

On DeepSeek V4.1-Flash the cached figure is AUD 0.0011 per call, or about 34 cents a day at the same volume.

Note what that spread tells you. The interesting variable is not long context versus retrieval. It is your cache hit rate. Caches on the major providers are time-limited and the write is charged at or above the normal input rate, so an automation that fires in a steady stream all day behaves completely differently to one that fires twice an hour. Twenty cache rewrites a day on Claude at that corpus size adds about AUD 76 daily — still cheap, but 2.7 times the cost of the reads themselves.

Output is not free either. At AUD 75.75 per million, an 800-token answer costs about six cents — comparable to the cached input cost of the whole corpus. Once context is cheap, verbosity becomes the expensive part.

Where the line sits

Long context beats retrieval when all of the following hold.

The corpus fits, with room to spare. A million-token window is not a million tokens of working space. You need room for the query, the conversation, tool outputs and the answer. Treat 300,000 to 400,000 tokens of source material as the comfortable ceiling for a 1M-token model, not 950,000.

The corpus changes slowly. If documents change weekly, you rebuild the prompt payload weekly and move on. If they change hourly — live inventory, open job status, today's pricing — you are back to fetching current state at call time, and that is retrieval by another name.

The volume is steady enough to hold a cache. Bursty, low-frequency use pays the write cost repeatedly and loses most of the advantage.

You need the whole corpus, not a slice. Retrieval's real weakness is that it can only reason over what it retrieved. Questions like "which of our contracts have a termination clause shorter than 30 days" or "does anything in the manual contradict this new procedure" require seeing everything at once. Retrieval systematically fails those, and no amount of chunking tuning fixes it.

Retrieval stays the right answer when the corpus runs to millions of tokens — every email thread since 2019, the full document management system, years of job history. It also stays right when you need hard filtering by customer, site or date before the model sees anything, which is a security control as much as a relevance one.

The honest middle case is a hybrid: the stable canon in cached context, a narrow retrieval call for live records, and a cheap classifier deciding which is needed. TypeSafe AI's Jev, reported on 17 September 2026 as a non-LLM decision model priced at US$42 per billion input tokens with free output, is the kind of component that makes that routing layer cost effectively nothing.

What you still have to engineer

Skipping the retrieval pipeline removes the vector store, the chunking strategy, the embedding refresh job and the retrieval evaluation harness. It does not remove the following, and this is where these projects actually consume effort.

Prompt assembly and versioning. Something has to turn a folder of Word documents, PDFs and a SharePoint library into a stable, ordered, deduplicated token payload — and rebuild it when the source changes, without silently dropping a document. That payload needs a version number, because when an answer is wrong you need to know which corpus produced it.

Cache discipline. Cache hits depend on byte-identical prefixes. A timestamp, a user name or a reordered document at the top of the prompt destroys the hit and multiplies your bill by forty. Getting the stable material first and the variable material last is the single highest-value design decision in the whole build, and it is the one most often got wrong. This is the sort of thing worth getting right at design time when you are building AI into existing business systems, because retrofitting cache-friendly prompt structure after launch usually means rewriting the integration.

Evaluation. Models do not attend evenly across hundreds of thousands of tokens, and position within the payload affects recall. You need a set of known-answer questions run against every corpus rebuild and every model change. With five labs shipping in ten days, model change is now a routine event rather than an annual one.

Access control. A cached corpus is one blob. If half the policy library should not be visible to casual staff, the context approach gives you no natural boundary — you need separate caches per audience, which multiplies the write cost and the maintenance. Retrieval handles this more gracefully.

Tool and agent safety. If the same system also calls accounting, CRM or file storage, the hardening matters. Microsoft's Agent Framework 1.19.0 for Python introduced per-invocation MCP-session scoping, identity and origin authentication, ZIP-only skill archives and archive-digest verification on 17 September 2026 — which tells you what the known failure modes are: credentials and state leaking between requests, and untrusted skill packages.

One caution on the prices themselves. Gemini 3.8 Flash's rate was announced as running to 31 December 2026, several of these figures come from secondary reporting, and the whole set has moved sharply twice this year. Design so the model is a configuration value, not an assumption baked through the code, and re-run the numbers each quarter.

The first move

Token-count your actual corpus before designing anything. Export the documents you would want the model to see, concatenate them, and divide the word count by 0.75 for a workable token estimate. Then count how many times a day a query would hit it. Those two numbers — corpus size and daily call volume — decide the architecture, and almost everyone guesses both wrong by an order of magnitude in one direction or the other.

Bring us those two numbers and we will run them properly: a costed comparison of cached-context versus retrieval for your specific corpus at your specific volume, the cache-write overhead included, and a plain answer on whether the workload is worth automating at all. Start with the ROI calculator and we will come back with a scope and a build estimate — typically two to four weeks of delivery — or a straight recommendation not to bother, which is sometimes the correct result.