Private AI's Real Cost: Power, Cooling and a Two-Year Timing Trap

Sixty-three percent of surveyed companies already use an open AI model, according to Linux Foundation research cited in a new industry analysis — and that shift is quietly turning enterprise infrastructure decisions into board-level bets.

The piece, a thought-leadership analysis on private AI, argues a simple point with expensive consequences: once you bring a model in-house, you inherit everything that used to be your cloud provider's problem. Power. Cooling. Compute. Placement. Connectivity. "If you size it wrong, you own capacity that doesn't fit the workload you actually end up with."

Why are companies pulling AI in-house?

Two forces drive the move. Regulated businesses — banks, pharma — were never going to put proprietary research or customer records in someone else's tenant. Everyone else is reacting to the inference bill at scale and to dependence on a single provider for something now core to operations.

The economics logic has also flipped. Open-weight models handle day-to-day work like summarizing documents, classifying tickets and answering questions against a company's own knowledge base. Frontier models get reserved for jobs that genuinely need them.

What does 'build the base, rent the spike' mean?

The recommended architecture mirrors long-standing compute practice. Build private capacity for steady, predictable weekly demand. Rent external capacity for spikes and heavy jobs. AI doesn't change the logic — it raises the density of the base and widens how far the peak can swing.

The hard part is making the two behave as one system. Both environments need access to the same data and the same controls, without rebuilding the pipeline every time a workload crosses over.

Where does the infrastructure actually go?

Buying GPUs is the easy call. Placing them is not. High-density AI hardware doesn't drop into a standard enterprise data center without major power and cooling upgrades, and some providers can't retrofit at all — their facilities were never built for that density.

Uptime Institute's 2025 AI Infrastructure Survey found 37% of respondents named power availability as a main factor in where they run AI inference workloads. Most companies land on a mix of colocation and cloud, and data usually decides: it's spread across locations, expensive to move, and in regulated industries some of it legally can't leave a country or environment.

The author frames this as the third great exchange. The internet needed connectivity; cloud needed fabric; AI needs both plus enormous power in the same place. "The companies that get private AI right are going to be the ones that put compute where power and connectivity meet instead of running those as two separate procurement exercises."

Why does agentic AI raise the stakes?

Agents move constantly. One agent calls several models, queries multiple systems, pulls data from wherever it lives and hands work to other agents. Every hop crosses a boundary, adding latency and raising access-control questions — sharpened when data-residency rules pin certain data to a country or private environment.

That's why infrastructure must be programmable: connections between models, data and environments should be configured, not engineered from scratch each time. Standards like the Model Context Protocol (MCP) point the way. An agent needing a new data source could request the connection like any other service, with policy controls intact. New models ship every few months; if each one triggers a rebuild, you've lost the flexibility you went private to get.

How do the economics work?

The core metric is blunt: power to tokens. Companies doing this well know exactly how much power they put in and how many tokens they get out, then plan headroom above that — because capabilities arrive quarterly and building to today's exact edge is a trap.

The GPUs are only part of the bill, too. Housing, cooling, powering and connecting them can match the hardware spend. And the skills gap is real: most enterprises have never operated high-density compute, and that expertise isn't hired in a quarter.

What's the timing problem?

A data center takes about two years to build. In two years, the models, hardware and applications will look considerably different — running workloads nobody can fully describe yet. The analysis's bottom line: avoid infrastructure commitments that remove your options before you've made the choice. Build for today's workload, keep connections programmable, and put the compute where fabric meets power.