Valkey 9.2 Cuts Inference Costs Up to 40% With Distributed KV Cache
Valkey 9.2 ships distributed KV caching that cuts inference spend up to 40% and first-token latency up to 80%. Multiverse Computing served 50% more requests on the same GPUs.
By Grace Kim
4 min read

What's News
- Valkey released version 9.2 on November 19, 2026 with distributed KV caching for AI inference.
- Valkey claims up to 40% lower inference spend and up to 80% lower first-token latency on existing GPUs.
- Multiverse Computing served 50% more requests on the same GPUs; peak-traffic first-token latency fell 95%.
- Valkey's benchmark on Llama-3.1-70B across 16 H100s measured ~25% more requests served for ~15% added GPU cost.
- LZ4 replication compression cut bandwidth 52% with under 1% throughput overhead on 3 million keys.
Valkey 9.2, released November 19, 2026, ships distributed key-value caching that the project says can cut AI inference spending by up to 40% and first-token wait times by up to 80% on existing GPU infrastructure.
The new cache addresses a specific inefficiency in how large models generate text. A technical post published October 20, 2026, describes a shared tier that sits beyond GPU and host memory and stores the key-value tensors a model produces when processing prompt instructions, tool definitions and conversation history. Inference engines can reuse those tensors instead of recomputing them during prefill — the processing stage that runs before any new output appears. Because the tier is shared across GPU servers, any node can load a conversation that its local memory has evicted.
That reuse is where the money is. Repeated context — system prompts, tool schemas, chat history — gets computed once instead of many times.
What do early users report?
Multiverse Computing's director of AI platform, Ennio Pastore, said distributed KV caching on Valkey let the company serve 50% more requests on the same GPUs while meeting its one-second target for median time to first token. Median first-token latency fell 23% at normal traffic and 95% at peak traffic. Multiverse also contributed LMCache connector improvements intended to make distributed caching more resilient.
Self-managed deployments running open-weight models connect the cache through the LMCache connector to engines including vLLM and SGLang with a configuration change — no application-code updates required. Version 9.2 also adds Path Hash, a radix-tree-backed data type supporting exact lookup, longest-prefix matching and prefix traversal. Valkey positions it as a lightweight index for locating the longest matching cached prefix and where it lives, while the bulkier AI data stays in GPU or host memory.
How big are the benchmark gains?
The synthetic benchmark in Valkey's technical post ran 100 concurrent conversations with 12,000 to 28,000 tokens of context, arriving at 8 requests per second across two GPU nodes with eight H100 accelerators each. The nodes served Llama-3.1-70B through vLLM and LMCache, backed by a three-shard Valkey 9 cluster in the same availability zone as the shared cache.
The results: about 25% more requests served and about 28% more output tokens, while the cache added roughly 15% to the GPU bill. Run the arithmetic and the throughput gain comfortably outruns the added cost — but only when context actually repeats.
Valkey's guidance draws the boundaries plainly. The shared cache offers little benefit when the working set already fits in GPU and host memory, or when requests lack repeated context. It also cannot conjure GPU compute when new-token prefill becomes the bottleneck. Recommended placement is the same availability zone as the GPU fleet, since the cache sits on the first-token path and cross-zone traffic adds both latency and transfer charges.
What else changed under the hood?
Version 9.2 also targets the datastore's own infrastructure costs. Opt-in forkless snapshots and full replica synchronizations run in a background thread instead of spawning a separate server process, avoiding copy-on-write memory overhead during write-heavy work. The feature costs four bytes per key, and the prefer-forkless policy falls back to fork when necessary.
Replication compression is negotiated separately with each replica, so older or opted-out replicas keep receiving plaintext. In a benchmark of 3 million keys of roughly 315 bytes each, LZ4 cut replication bandwidth by 52% with less than 1% throughput overhead. Whole-stream snapshot compression supports LZ4 and Zstandard; Zstandard requires a build-time option and targets cross-region replication, where network costs run high relative to compute.
Large sorted sets now use a B+ tree instead of a skiplist, cutting per-member memory overhead 36% to 43% in a test with 5 million members containing 20-byte elements. The release also adds server-side hot-key detection, reusable access-control roles, priority handling for cluster heartbeats and replication, reserved administrative connections, and optional write throttling when a replica falls behind.
The 9.2.0 release record marks this as the first stable 9.2 release, with Docker tags and platform binaries on the download page. Valkey calls 9.2 a minor release without breaking changes, with one caveat: append-only file bases written with LZ4 or Zstandard compression require a compatible Valkey version if an operator later downgrades.
The release ties together two sides of inference economics — recovering GPU capacity lost to repeated prefill and trimming the data layer's memory and network overhead. For operators weighing the new tier, the documented results anchor its value to workload reuse and eviction pressure, with a hard ceiling once new-token prefill already saturates available GPU compute.
Source: unite.ai
More from Grace Kim
Show full bio
Market editor covering people, launches and day-to-day practice in entrepreneurship, startups and venture funding at Business Bearings.
2,447 articles