Innovation & Tech

AWS Collapses 20 Inference Endpoints Into One With Global Gateway

AWS's SageMaker HyperPod Global Inference Gateway, launched November 20, 2026, routes inference across Regions from one endpoint using four weighted signals, with automatic failover and PrivateLink-only traffic.

By Olivia Hart

4 min read

AWS Routes HyperPod Inference Across Regions From One Endpoint
AWS Routes HyperPod Inference Across Regions From One Endpointblickpixel / pixabay

What's News

  • AWS launched SageMaker HyperPod Global Inference Gateway on November 20, 2026, in 13 Regions.
  • In AWS's example, four models across five clusters need 20 endpoints per-model; the global gateway provides one.
  • Cluster selection uses four signals with defaults: RegionProximity 0.40, QueueDepth 0.25, KVHeadroom 0.20, ErrorRate 0.15.
  • AWS states the gateway does not log or store prompts or completions and routes only over PrivateLink.
  • Failed requests retry once on another cluster; with no healthy cluster, the gateway returns HTTP 503.

Amazon's new SageMaker HyperPod Global Inference Gateway, launched November 20, 2026, cuts a 20-endpoint inference setup to a single managed endpoint across Regions. In AWS's own example — four models served across five clusters in two Regions — per-model serving demands that clients track 20 endpoints. Cluster-level gateways reduce that to five. The global gateway reduces it to one.

The service targets a straightforward capacity problem: requests queue on a saturated cluster while already-provisioned GPUs sit idle elsewhere. By putting proximity, capacity and failure signals in one policy, the service makes idle GPU capacity a configuration decision rather than a separate networking project. The managed tier also removes the need to build and maintain load balancers, DNS and cross-Region networking, and lets peak traffic overflow to clusters with spare capacity.

How does cluster selection work?

The launch adds a second routing tier above the existing SageMaker HyperPod Inference Gateway. The cluster-level gateway keeps selecting individual model pods using signals such as key-value (KV) cache utilization and queue depth. The new global tier selects the cluster first and leaves pod selection unchanged.

For each request, the gateway scores healthy clusters on four configurable signals:

  • KVHeadroom — available KV cache and GPU memory, default weight 0.20
  • QueueDepth — how backed up the cluster is, default weight 0.25
  • RegionProximity — distance between cluster and gateway Region, default weight 0.40
  • ErrorRate — recent failed requests, default weight 0.15

Customers overriding the defaults must set all four weights, from 0.0 to 1.0, totaling 1.0; a zero disables a signal. UpdateGlobalInferenceGateway changes weighting for new requests without reprovisioning, while in-flight requests finish under old settings. AWS says workloads that prioritize resilience over proximity can raise ErrorRate and lower RegionProximity — though every claim of tunability rests on the operator setting the weights correctly.

What happens on failure?

An operator publishes each participating cluster by enabling spec.globalRouting in the existing InferenceGatewayConfig, then calls CreateGlobalInferenceGateway with the clusters to group. Publishing creates a Network Load Balancer, a VPC endpoint service and certificate resources in the customer's account. An unpublished cluster cannot be added.

Clients send OpenAI-compatible requests signed with AWS Signature Version 4 to the regional endpoint. The gateway authenticates the caller and selects a healthy cluster; AWS PrivateLink carries the request within or across Regions, and the cluster's Inference Gateway chooses the model pod. SageMaker operates the routing fleet, private endpoints, certificate validation, health checks, authentication and rate limiting. A service-managed publisher on each cluster supplies load metrics.

On failure, the gateway shifts traffic to the next-best cluster without client changes and brings a recovered cluster back gradually. A failed request is retried once on another cluster unless streaming has started; if no healthy cluster remains, the gateway returns HTTP 503. Adding a healthy cluster changes neither the endpoint nor other clusters. Removing one stops new requests but lets in-flight work finish.

IAM separately controls standard and streaming invocation. Customer JSON Web Tokens, when used, travel inside the SigV4 signature. Requests reach clusters only over PrivateLink — and AWS states the gateway does not log or store prompts or completions, a privacy claim customers must take on the company's word.

Bagelcode tested the failover claim. VP of AI Platform Jinhyung Park said in AWS's announcement that beta tests routed requests across two Regions through the gateway and continued serving them when one cluster could no longer serve the model. That gave the company's small platform team cross-Region failover without building or maintaining its own routing layer.

Where is it available, and who should migrate?

The service runs in 13 Regions: US East (N. Virginia and Ohio), US West (N. California and Oregon), South America (São Paulo), Europe (Frankfurt, Ireland, London and Stockholm), and Asia Pacific (Tokyo, Mumbai, Singapore and Sydney).

AWS advises single-cluster users to stay on the cluster Inference Gateway endpoint and adopt the global service only when adding a second cluster. Existing per-cluster endpoints keep serving during migration, and a gateway containing only clusters in one Region can keep a workload there. One cost trap deserves attention: a published cluster left outside any gateway receives no traffic even though its Network Load Balancer continues to run and incur charges. Deleting the global gateway stops its endpoint but leaves per-cluster endpoints serving.

Source: unite.ai

Share this article:

More from Olivia Hart

Olivia Hart

Show full bio

Staff writer covering people, launches and day-to-day practice in entrepreneurship, startups and venture funding at Business Bearings.

2,535 articles

Related articles

« Previous article