AWS Adds KV Cache-Aware Routing for Amazon SageMaker AI
AWS launched KV cache-aware routing for SageMaker AI on November 17, 2026, citing 97% lower median time to first token and 234% higher throughput, with 14 to 19 milliseconds of overhead in one test.
