AWS Adds NVIDIA GPU Partitioning to SageMaker Endpoints

By Olivia Hart

AWS reported 53% to 89% more output tokens per second from a mig-3g.40gb NVIDIA H100 partition than a whole A100-40GB in selected Amazon SageMaker Inference tests. Amazon Web Services (AWS) announced NVIDIA Multi-Instance GPU (MIG) support for Amazon SageMaker Inference on November 6, 2026.

What does GPU partitioning change?

Before MIG, AWS said accelerator count capped how many models an instance could host, regardless of memory or compute demand. Customers considering a newer GPU also had to allocate the entire device.

AWS’s MIG option gives each partition fixed compute units and dedicated memory with hardware isolation. AWS said resident models avoid the first-use loading delay that a multi-model endpoint can incur when placing a model on its shared device.

AWS supports partitioning only for real-time endpoints that use inference components, SageMaker’s construct for deploying several models to one endpoint and assigning compute. AWS excludes single-model endpoints, serverless endpoints, heterogeneous instance pools and asynchronous workloads; async invocation cannot address an inference component. AWS therefore limits the feature to a specific multi-model, real-time architecture.

What did Salesforce expect from the feature?

Richard Chang, a Salesforce software engineering architect, said the company expected MIG to “consolidate workloads safely, improve GPU utilization and reduce inference costs while preserving predictable performance and isolation.” Salesforce helped AWS define the requirements and validate the initial implementation, according to AWS.

Salesforce is preparing to validate the software with representative production workloads. Until those tests report results, Salesforce’s projected savings remain unproven.

How does AWS configure partitioning?

AWS’s developer documentation defines two optional fields. AcceleratorPartitionLayouts declares the carve on a production variant, while AcceleratorPartitionType selects an inference component’s profile. AWS applies the selected layout uniformly to every accelerator on every instance in the variant.

Omitting both fields preserves existing endpoint behavior. AWS adds no API operations, resource types or AWS Identity and Access Management actions.

AWS’s example divides each of eight A100s on an ml.p4d.24xlarge into mig-4g.20gb, mig-2g.10gb and mig-1g.5gb partitions, producing 24 partitions across the endpoint. Each inference-component copy occupies one requested partition, while InvokeEndpoint continues to address models by component name without a partition identifier. AWS said carving the accelerators added roughly 17 seconds to provisioning.

GPU isolation covers compute and memory, not the entire host; models still share host CPU, host memory, storage, networking and endpoint routing. AWS rejects a component that requests a profile absent from the layout, rather than assigning it a whole GPU.

Changing a running endpoint’s layout requires a new endpoint and traffic migration, and moving a component to another profile requires a new component. AWS’s setup is flexible at creation but inflexible after deployment.

What did the benchmark show?

AWS compared one mig-3g.40gb H100 partition with a whole A100-40GB using Llama-3.1-8B-Instruct on production SageMaker real-time endpoints. Each setup used one model replica, identical containers, tokenizers and client configurations, while genai-bench drove five traffic shapes at concurrency levels from 2 to 512.

According to NVIDIA’s profile guide, the tested H100 profile receives three of seven streaming-multiprocessor shares and four of eight memory units. AWS said the two configurations offered comparable memory capacity but different compute allocations.

At the load where each scenario’s cost benefit was largest, AWS reported 53% to 89% more output tokens per second from the H100 partition. AWS also reported 8.9% to 42.1% lower inter-token latency and 5.6% to 47.5% lower time to first token.

At the highest concurrency on 6,000-token and 8,000-token workloads, AWS said the A100 missed the serving deadline on one-third to one-half of requests, while the partition failed none. AWS’s benchmark supports the partition under the tested load, not as a blanket replacement for a full GPU.

When does partitioning cost less?

AWS presented its cost analysis as a case-by-case comparison rather than a universal verdict. Once traffic passed a workload-specific break-even point, AWS found 18% to 34% lower cost per million output tokens; break-even occurred at concurrency 64 for 100-token prompts and concurrency 4 for 8,000-token prompts.

Below those points, AWS said the whole A100 remained cheaper, leaving short-input or low-concurrency workloads potentially better suited to the older GPU. AWS calculated unit cost by dividing on-demand instance prices by the number of tested units on each instance. AWS’s practical takeaway is simple: actual traffic determines whether partitioning saves money.

Which hardware and regions are available?

AWS launched the feature across four instance families, with partition counts and profiles determined by each accelerator. The supported hardware is:

• P4d: NVIDIA A100, 40 GB GPU memory; maximum 7 partitions per GPU.
• P5: NVIDIA H100, 80 GB GPU memory; maximum 7 partitions per GPU.
• G7: NVIDIA RTX PRO 4500 Blackwell, 32 GB GPU memory; maximum 2 partitions per GPU.
• G7e: NVIDIA RTX PRO 6000 Blackwell Server Edition, 96 GB GPU memory; maximum 4 partitions per GPU.

AWS offers GPU partitioning in US East (N. Virginia), US East (Ohio), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Tokyo), Europe (Frankfurt) and Europe (Ireland), subject to regional instance availability and endpoint quota. No partition spans accelerators, so models requiring a full GPU or multiple GPUs still need whole devices.

Customers continue paying for the full GPU instance; partitioning increases usable capacity but does not reduce the hourly price. AWS adds capacity without an instance-price discount.

What remains invisible and what setup does AWS require?

AWS’s launch turns each supported accelerator into several fixed GPU sizes. Partitioned endpoints and their inference components do not publish GPU, CPU or memory utilization metrics to CloudWatch, although invocation and latency metrics remain available.

On P4d, the mig-1g.10gb profile requires NVIDIA’s R525 driver or newer. AWS directs customers to a newer inference AMI because the default image does not provide it.

AWS leaves capacity planners with full-instance billing and fewer utilization metrics. The next proof point is Salesforce’s planned validation on representative production workloads; until then, AWS’s own break-even data makes traffic profiles the decisive purchasing test.