Innovation & Tech

Redis Cache Cut This AI Chatbot's Token Use 4.5-Fold

A Redis cache cut one AI chatbot's token consumption from 23,000 to 5,000 per request — 4.5 times cheaper — before RAG took over the rephrased questions a cache cannot match.

By Olivia Hart

4 min read

This Is the Reason Your AI Tools Actually Remember Anything
This Is the Reason Your AI Tools Actually Remember AnythingTumisu / pixabay

What's News

  • A Redis cache cut Rune Score's most expensive step from 23,000 to about 5,000 input tokens — a 4.5-fold reduction.
  • A RAG hit reduced a request from 20,000 to 8,000 tokens, a 60% cut versus roughly 80% for an exact cache hit.
  • Rune Score, a World Cup sports analytics chatbot, was built as a distributed system handling costs, memory and fallbacks.
  • Redis's engineering blog reports combined retrievers surface more relevant results than any single method alone.

A Redis cache cut the most expensive step in one AI chatbot's pipeline from 23,000 input tokens to about 5,000 — a 4.5-fold reduction, according to the developer's own documented measurements on LinkedIn. The numbers come from Rune Score, a sports analytics chatbot built during this year's World Cup. Behind its simple chat interface sits a distributed system juggling data freshness, user rephrasing, extraction failures, LLM call costs, streaming, memory, fallbacks and security. Its builder's core finding: most AI products waste money by repeating work, and the cheapest fix is not the most fashionable one.

Why the cache came first

The developer's first move was not a vector database. It was a Redis cache governed by a simple rule: identical requests get an identical key, such as the football matchup Argentina vs. Spain. The developer wrote: "without the cache, a fresh extraction followed by an analysis can consume up to 23,000 input tokens: scraped content, context, history. With a cache that hits the same key, that number drops to about 5,000 tokens."

The reasoning is arithmetic, not architecture. If 10 people ask the same question within a short window, rerunning the full pipeline 10 times burns money for no reason. A cache that teaches you what users actually ask for costs almost nothing to run. The takeaway is plain: start simple, because the cheap tool shows you the demand pattern before you spend on anything advanced.

Where exact matching breaks

The weakness appears immediately in real usage. People do not speak in fixed labels. One user writes "FC Barcelona vs Real Madrid"; another writes "Barça – Madrid." A strict cache treats each phrasing as a brand-new request. The result is duplicate work — or a rerun of the entire expensive pipeline because the wording shifted by one word. The answer was already stored; the cache simply could not recognize the question.

This is where retrieval-augmented generation, or RAG, earns its place. AWS defines RAG as "a technique used to augment a large language model (LLM) with external data, such as a company's internal documents." Instead of matching an exact label, RAG searches a broader knowledge base by meaning. Google Cloud's documentation explains the mechanism, embeddings: "the closer objects are in the embedding space, the more semantically similar they are, and greater distances indicate their greater semantic dissimilarity." In plain terms, the system retrieves a stored answer even when the question is worded completely differently.

What actually breaks quality

The developer's harder lesson concerns inputs, not retrieval technology. Citing the classic computing principle that TechTarget defines as garbage in, garbage out — "the idea that in any system, the quality of output is determined by the quality of the input" — the builder reports that every shortcut on content extraction visibly degraded the downstream analysis.

Ranking is equally fragile when it leans on one signal. Redis's own engineering blog states: "the consistent takeaway across these studies is that combining retrievers tends to surface more relevant results in the top positions than either method on its own." Merging exact keyword matches, meaning-based search and recency into one ranking beats any single measure. The takeaway: the bottleneck in a RAG system is usually what feeds it, not the retrieval layer itself.

Is the flexibility worth the cost?

The measured trade-off is concrete. A RAG hit in Rune Score brings a request from 20,000 tokens to 8,000 — a 60% reduction, against roughly 80% for an exact Redis hit. RAG is slower and less economical per request. What it buys instead is tolerance for how people actually phrase questions, memory that outlives a cache's expiration window, and an architecture reusable beyond a single product. That is a defensible trade, provided the cheaper option has already proven insufficient.

The developer's bottom line is sequential, not either-or: build the cheap cache first, let it reveal real demand, and only then invest in full retrieval for the cases a cache structurally cannot handle. The final caveat deserves weight — the best refactor is not the theoretically right one, but the one you can roll back in seconds when the metrics fail to hold.

Original: redis.io

Share this article:

More from Olivia Hart

Olivia Hart

Show full bio

Staff writer covering people, launches and day-to-day practice in entrepreneurship, startups and venture funding at Business Bearings.

2,535 articles

Related articles

« Previous article