AWS Turns SageMaker HyperPod Node Software Into Managed State
AWS launched managed cluster extensions for SageMaker HyperPod Slurm on November 20, 2026, replacing lifecycle scripts with declarative node-software state for observability and user management.
By Daniel Okafor
4 min read

What's News
- AWS launched managed cluster extensions for SageMaker HyperPod Slurm on November 20, 2026.
- The first two extensions, sagemaker-observability and sagemaker-user-management, carry no additional charge.
- Migration requires the November 2, 2026 HyperPod AMI or later and manually removing script-installed agents during a maintenance window.
- Extensions report Creating/Updating, Active or Degraded states, with node-level issues grouped by cause.
Amazon Web Services launched managed cluster extensions for Amazon SageMaker HyperPod Sluml on November 20, 2026, letting administrators install, update and remove node software through APIs or the SageMaker AI console instead of lifecycle scripts. The first two extensions cover observability and user management. Both are free, though customers still pay for the underlying resources they touch.
The feature targets the part of a shared AI training cluster that keeps changing after provisioning. During 2026, HyperPod had already added API-driven Slurm configuration, AMI-based node setup, continuous provisioning, automatic topology management and deep health checks. Those tools manage the cluster's definition, capacity, network layout and hardware health. Cluster extensions apply the same declarative approach to node software that shifts as research teams, monitoring requirements and package versions evolve.
How does the mechanism work?
Administrators manage the AWS-managed packages through the console, AWS CLI or an SDK. HyperPod validates settings against the selected extension version's schema before rollout, then records the requested version and settings as the desired state. A node agent applies only the components each node's role requires. Nodes added later, through scaling or replacement, inherit the same configuration automatically.
Updates are surgical. Only the changed settings get submitted; HyperPod merges them into the stored configuration and restarts only the affected components. Nodes stay in place. Lifecycle scripts do not rerun. New extension versions can be installed without an AMI update, and HyperPod re-checks stored settings against the new version's schema before rollout. Upgrades only move forward.
What do the health signals show?
Each extension reports one of three states during its lifecycle:
- Creating or Updating during rollout
- Active when every node runs the requested configuration without a reported problem
- Degraded when nodes fail or report problems
A DescribeClusterExtension call groups node-level health issues by cause, with a code, a message and an affected-node count. HyperPod retries its own internal failures automatically. Administrators can fix other causes and retry affected nodes with an update call that contains no configuration changes.
What do the two extensions actually do?
The sagemaker-observability extension collects node, GPU, Elastic Fabric Adapter, Slurm job and Slurm cluster metrics, and ships them to Amazon CloudWatch or Amazon Managed Service for Prometheus. Metric categories, levels and collection intervals are configurable. Nodes run only role-relevant components — DCGM and EFA exporters on GPU nodes, Slurm cluster metrics on the controller. An optional Amazon Managed Grafana workspace adds pre-created Clusters and Jobs dashboards that tie cluster-wide health to per-node infrastructure metrics and per-job resource use, including CPU, memory, GPU utilization, disk I/O and NCCL data.
The sagemaker-user-management extension creates Linux users and groups with consistent IDs across nodes, controls SSH and passwordless sudo access by node type, and registers users with Slurm accounting. Accounts can be declared locally, drawn from Active Directory over LDAPS, or both — including connections to AWS Directory Service for Microsoft Active Directory. Administrators review access through the API instead of signing into each node.
What does migration require?
Cluster extensions need a HyperPod Slurm cluster running the November 2, 2026 HyperPod AMI or later, with continuous provisioning enabled. Because extensions run software as root, AWS treats create, update and delete permissions as administrator permissions and advises against granting them to the cluster execution role. That role still needs access to metric destinations and Active Directory secrets. User management requires a shared file system for home directories. Separate extension ARNs let platform and monitoring teams own different extensions.
Existing clusters migrate one capability at a time, with scripts and extensions coexisting. The sequence: update the cluster software, disable that capability in lifecycle scripts, manually remove script-installed agents during a maintenance window, then create the extension on the clean cluster. Each component needs exactly one owner — managing the same exporter, user or service through both mechanisms can cause installation or runtime conflicts. Custom AMIs and lifecycle scripts remain available for organization-specific software, and administrators keep SSH and direct configuration access to every node, including the controller.
Both extensions are available in all Regions where SageMaker HyperPod is supported. For shared foundation-model clusters, the practical shift is that common node software becomes declared, continuously checked cluster state rather than a pile of provisioning scripts — and after the one manual cleanup step, every replacement node arrives already compliant.
Source: unite.ai
More from Daniel Okafor
Show full bio
Correspondent covering money, funding and market shifts around entrepreneurship, startups and venture funding at Business Bearings.
2,538 articles