AWS Adds KV Cache-Aware Routing for Amazon SageMaker AI
AWS launched KV cache-aware routing for SageMaker AI on November 17, 2026, citing 97% lower median time to first token and 234% higher throughput, with 14 to 19 milliseconds of overhead in one test.
By Amara Osei
4 min read

What's News
- Amazon Web Services launched KV cache-aware routing for Amazon SageMaker AI on November 17, 2026.
- In long-prompt tests, AWS reported up to 97% lower median time to first token and 234% higher throughput than random routing.
- Cache hit rates increased from 3–48% to 80–93% in 30-minute tests using seven ml.p5.48xlarge instances.
- A two-instance Qwen3-4B-Instruct-2507 comparison measured 14 to 19 milliseconds more CloudWatch OverheadLatency.
- Launch requires vLLM 0.33.0 or later or Amazon Linux server-sagemaker-cuda-v2.8.0 or later, plus at least two instances or two inference-component copies.
By Amara Osei
AWS reports up to 97% lower median time to first token and 234% higher throughput than random routing in long-prompt tests under load. Amazon Web Services introduced the feature for Amazon SageMaker AI real-time endpoints on November 17, 2026.
AWS measured a cost in a separate two-instance Qwen3-4B-Instruct-2507 comparison. Average CloudWatch OverheadLatency was 14 to 19 milliseconds higher with KV-aware routing, even as model latency and time to first token improved; AWS defines the metric as time spent outside the model container.
AWS says the routing lookup adds tokenization and index-processing overhead: a few milliseconds for short prompts and more for longer ones. For standard text-only requests, the service passes computed tokens to the model, avoiding a second tokenization.
What did AWS measure?
AWS’s 30-minute tests ran Llama 3.1 70B Instruct on seven ml.p5.48xlarge instances. AWS’s test conditions covered 20 or 100 simulated users, multi-turn chats, document question-answering sessions and prompts reaching about 64,000 tokens.
AWS reports that cache hit rates rose from 3–48% with random routing to 80–93% with KV-aware routing. The company calls the results workload-dependent and says short prompts gained least because they leave less prefill computation to reuse.
How does the routing work?
AWS says the KV cache stores prior prompt computations in GPU memory, allowing a model to reuse matching input tokens instead of repeating the initial processing, or prefill. AWS says the feature uses cache-state events reported by the model container to place requests on instances likely to hold those computations.
AWS targets workloads where misplaced requests waste GPU work: chat assistants, retrieval-augmented generation and agents with long system prompts. AWS says sending a request elsewhere can force an instance to recompute context that another GPU already holds.
SageMaker AI’s existing prefix-aware strategy infers cache location from a fixed request prefix. AWS says it cannot detect evictions, and early turns in a growing conversation can hash differently and land on separate instances.
KV-aware routing instead consumes events from the model container whenever cache entries are stored or evicted. AWS says the service maintains a managed index of one-way truncated hashes mapped to instances or inference-component copies, without storing prompts, token IDs, request payloads or KV tensors.
For each incoming prompt, AWS says the endpoint tokenizes and matches it against the index. It weighs the longest cached match against instance load and skips candidates that are too busy.
If lookup is slow, no useful match exists or all matching instances are busy, AWS routes the request through least-outstanding-requests routing. AWS says the service still serves the request in that case.
AWS says scale-in removes index entries for removed instances, while new or replacement instances start with empty caches. That creates temporary cache-rebuild and cold-start latency considerations.
What does deployment require?
At launch, AWS requires one of two container configurations:
- The AWS vLLM Deep Learning Container for SageMaker AI with vLLM 0.33.0 or later.
- The
Amazon Linux server-sagemaker-cuda-v2.8.0build or later.
AWS requires customers to enable prefix caching and KV cache events. They must set SM_VLLM_KV_EVENTS_CONFIG on the model and RoutingStrategy to KV_AWARE in the production variant, with no changes to client calls.
AWS supports GPU-backed real-time single-model and inference-component endpoints, including dynamic Low-Rank Adaptation adapters. Routing requires at least two instances for a single-model endpoint or two inference-component copies; with one destination, the setting has no routing effect.
AWS requires requests to use OpenAI-compatible Completions or Chat Completions payloads through supported invocation APIs. The company does not support InvokeEndpointWithBidirectionalStream.
How visible are routing decisions?
AWS launched the feature without dedicated metrics or customer-account routing-decision logs. The SageMaker AI console and its metrics do not expose routing fallbacks.
AWS says customers can inspect usage.prompt_tokens_details.cached_tokens after enabling prompt-token details. They can also monitor container metrics, including cache hit rate and time to first token, through detailed observability and CloudWatch.
AWS says the feature is available in all commercial AWS Regions that support SageMaker AI real-time inference with a supported container. AWS says there is no additional charge for the feature beyond the endpoint’s instances.
AWS still describes prefix-aware routing as the simpler option for stable shared prompts, short contexts and moderate load. AWS says KV-aware routing addresses longer reusable context, memory pressure and frequent eviction. The next deployment decision is workload-specific: compare recovered GPU prefill work with measured routing overhead and allow for cache rebuilds and cold starts after scaling.
Source: unite.ai
More from Amara Osei
Show full bio
Senior reporter covering companies, deals and strategy in entrepreneurship, startups and venture funding at Business Bearings.
2,248 articles