Google Cloud makes Pathways generally available on GKE

Over the last 90 days, Google Cloud said that “leading AI companies and global enterprises have processed more than 130 million Cloud TPU chip-hours using Pathways on GKE, across a broad range of TPU generations.” Google Cloud made Cloud Pathways generally available on Google Kubernetes Engine (GKE) in the same announcement.

Built originally by Google DeepMind to train and serve Gemini, Pathways is now a core runtime pillar of Google’s AI Hypercomputer stack. Google said it targets workloads spanning thousands of heterogeneous TPU chips.

How does the single controller work?

Pathways replaces the default multi-controller JAX execution model. Under that model, every TPU host runs its own Python interpreter. Hosts synchronize through collectives such as AllReduce.

Google built the single-controller system on the open-source Interim Framework Runtime (IFRT). Setting JAX_PLATFORMS=proxy with pathwaysutils lets one JAX client process view and orchestrate multiple large TPU slices as if they were locally connected.

Why did Google change the execution model?

Google said SPMD works well for uniform workloads but breaks down when jobs become heterogeneous. Pipeline parallelism can leave accelerators idle in pipeline bubbles. Reinforcement-learning loops for the sampler, reward model and trainer cannot overlap efficiently because no single process sees the full dependency graph.

Google also said one host failure could stop the entire collective and force a restart from checkpoint. That failure pattern interrupted Gemini 2.5 training multiple times per hour.

Pathways decouples the client from physical hardware. The client writes against virtual devices. A resource manager maps them to real chips, then handles placement, tensor movement and scheduling. It can remap work onto healthy chips after a host failure, which Google calls the basis for elastic training.

What ships with general availability?

Google Cloud’s GA release includes four headline features:

• Single-controller orchestration: One client manages graph evaluation, persistent Cloud Storage compilation caching and asynchronous XLA HLO dispatch across TPU slices.
• Colocated Python sidecars: Client-side containers with shared-memory support run compute-intensive Python logic, including data loading and checkpointing, directly on TPU hosts.
• Resilient and elastic training: Host-memory snapshots let the runtime dynamically reshard training down to N-1 slices during hardware failures without crashing the client.
• Interactive mode and Shared Pathways Service: Teams can run and debug JAX workloads from a notebook, or share a long-running pool of TPU slices across multiple users and jobs on demand.

Google Cloud said the product is generally available in every Google Cloud region supporting Cloud TPU v5e, TPU v5p, Trillium (TPU v6e) and Ironwood (TPU 7x) on GKE. Versioned images at us-docker.pkg.dev/cloud-tpu-v2-images/cloud-pathways support JAX 0.10.1 and later.

A Preview integration with Cluster Director extends Cloud Pathways to Slurm clusters. It supports unified multi-slice JAX execution, elastic training and interactive notebooks. Google Cloud said the Slurm preview uses container images tagged for JAX 0.11.1.

What does customer adoption show?

Google described one of the world’s largest consumer technology companies as running production batch and reinforcement-learning inference across tens of thousands of TPU v5p, Trillium (v6e) and Ironwood (TPU7x) chips. That customer consumes more than 100 million chip-hours per quarter. It also co-designed elasticity primitives with Google to expand into pre-training.

According to Google, ZenteiQ uses Pathways on GKE with Cluster Toolkit (gcluster), colocated sidecars and host snapshotting. The company uses the setup to scale elastic pre-training with MaxText and single-controller RL with MaxText and Tunix. The work spans multi-slice Trillium clusters for scientific foundation models.

Founder Prof. Sashikumaar Ganesan said the combination “pushes recovery time down to a minimum when disruptions occur, keeping our available TPU capacity fully utilized with very little stall time.” Google’s announcement provides no independent verification for the chip-hour figures or customer characterizations.

How does Pathways fit with Ray?

Google explicitly positions Pathways as complementary to Ray, not a replacement. Ray orchestrates Python tasks and actors across a cluster, treating accelerators as resources to schedule. Pathways operates at the execution layer inside the training step. It turns multi-program, multi-data (MPMD) workflows into one dataflow graph with a central scheduler.

In principle, Ray can orchestrate the pipeline while Pathways controls how each step executes on the chips.

Can teams outside JAX and TPU environments use it?

Pathways runs JAX programs on Google TPUs. iMasters said it is not a drop-in substitute for PyTorch DDP/FSDP on GPU clusters. That limits immediate adoption to teams already invested in, or evaluating, the JAX and TPU ecosystem.

iMasters said the underlying diagnosis applies broadly. Blocking collectives, all-or-nothing failure semantics and distributed debugging remain fundamental bottlenecks at scale. It described remapping around a failed host without a full restart as a tangible operational differentiator for multi-day runs on large clusters.

What remains unproven?

Google has not published independent benchmarks comparing Pathways throughput or recovery time with multi-controller JAX or PyTorch equivalents on comparable hardware. The 130 million chip-hour and 100 million chip-hour-per-quarter figures are self-reported aggregates. They lack breakdowns by workload type, model size and utilization efficiency.

The Cluster Director and Slurm preview has no stated GA timeline. Until those benchmarks appear, teams can test Pathways now, but they cannot yet validate its claimed operational advantage on comparable hardware.