Economy & Policy

California Proposes AI Kill Switch and 48-Hour Incident Reporting

California proposes state power to shut down frontier AI models immediately, continuous oversight by licensed verifiers, and a 48-hour critical incident reporting deadline.

By Olivia Hart

4 min read

Newsom Proposes Mandatory Frontier AI Oversight and Shutdown Powers
Newsom Proposes Mandatory Frontier AI Oversight and Shutdown Powersanalogicus / pixabay

What's News

  • Newsom released the legislative recommendations on November 18, 2026, based on a report dated November 16, 2026.
  • Critical incident reporting deadlines would drop from 15 days to 48 hours; imminent-catastrophic-risk reports stay at 24 hours.
  • Emergency shutdown orders would take effect immediately and remain in force during judicial review.
  • Models capable of catastrophic-risk actions would be covered even below the 10^26 operations training threshold.
  • A surcharge on verification fees would fund public CalCompute capacity reserved partly for independent safety research.

California Governor Gavin Newsom released legislative recommendations on November 18, 2026 that would let state officials order immediate emergency shutdowns of frontier AI models and impose continuous oversight by licensed independent verifiers on large developers.

The proposal comes from the California Government Operations Agency's report, AI Safety in 2026 and Beyond, dated November 16, 2026. It is intended to inform legislation when lawmakers return in December 2026. The recommendations are not yet enacted obligations — a caveat that matters, because the state's licensed verification sector barely exists today.

What would change for frontier developers?

The plan would build on California's existing requirement that large frontier developers publish safety frameworks. It adds state-set baselines for safety, security and transparency throughout training, internal use and deployment — roughly triple the lifecycle coverage of the current disclosure regime.

Every large frontier developer would face ongoing oversight by licensed independent verification organizations, or IVOs. These organizations would translate state standards into technical tests and update them as AI evaluation methods evolve. They would also gain access to models, personnel, systems, training environments and records they judge necessary, then send unredacted assessments to the state.

The Government Operations Agency would license IVOs and could revoke licenses for insufficient expertise, independence or rigor. If an assessor could not attest that a developer met the standards, the state could order corrective action after engaging with the company. Developers could challenge such determinations in court before they take effect — except under the expedited emergency process.

A surcharge on verification fees would finance publicly operated CalCompute capacity, with some of that capacity reserved for independent safety research and public oversight.

Takeaway: developers would move from publishing policies once to being audited continuously by state-licensed outsiders.

How would the kill switch work?

Each large frontier developer would maintain a Risk Containment Protocol, tested and verified on an ongoing basis by an IVO and certified as effective by a senior company official. The required capability would span compute, agents, internal access and model deployment rather than depend on a single on-off switch. Protocols would identify responsible personnel, coordinate with cloud providers and power suppliers, and define when a model can safely be restored.

Testing would address ways shutdowns can fail — including a model copying its weights to obstruct containment, or an outside actor seizing shutdown controls. For open-weight developers, containment requirements would apply only to deployments under their control, since they cannot shut down copies hosted by others. That concession is notable: it leaves open-weight releases substantially outside the shutdown regime.

State officials could order immediate shutdown upon credible reason to believe an imminent catastrophic risk or major loss-of-control event exists and the developer refuses to contain the danger. The order would require the narrowest workable intervention, take effect immediately and remain in force during judicial review, though the developer could seek a temporary restraining order.

Takeaway: shutdown orders would bite first and be litigated second.

What happens to incident reporting?

The reporting regime would cover attempted model-weight theft even without death or injury, all internal and external loss-of-control incidents, and a new near-miss channel. The report proposes cutting the deadline for most critical incident reports from 15 days to 48 hours, while retaining the 24-hour requirement for imminent catastrophic risks. Whistleblower protections would extend to former employees and contractors.

Models capable of actions that could create catastrophic risk would be treated as frontier models for reporting purposes even when trained with fewer than 10^26 integer or floating-point operations. That capability-based trigger addresses the report's stated concern that relying on training scale alone has left dangerous incidents outside state reporting requirements.

What about AI building AI?

Companies automating AI research and development would report quarterly how their compute is divided among training, research experiments, inference for AI-research agents, safety and security work, external customers and unattributed activity, with IVOs verifying the categories. Additional disclosures would cover training-data volume, cost, subject domains, human-versus-AI provenance, reinforcement-learning inference compute, rewarded misbehavior and mitigations applied. The report says these measures would show whether AI is taking over more of its own development and whether safety investment is keeping pace.

Recursive self-improvement — the strongest form involving an AI system independently developing its successor without human input — would face standby cap authority rather than an immediate limit. Officials could impose a time-limited, periodically reviewed cap on automated AI research when it exceeds a developer's committed thresholds or advances faster than its safeguards, creating material risk of catastrophic harm. The cap would lift once an IVO verified adequate safeguards.

The recommendations answer Newsom's September 18, 2026 executive order on independent oversight and an AI kill switch, which requested expert proposals within two months and accelerated implementation of California's independent-verification and AI-auditor laws. The Government Operations Agency developed the report with the California Office of Emergency Services and outside advisers Jason Goldman, Gillian Hadfield, Alondra Nelson and Rob Reich.

For developers, compliance would extend beyond publishing safety policies to continuous external scrutiny, faster incident reporting and demonstrable shutdown capabilities. The proposal leaves lawmakers with two connected decisions: how much intervention authority to enact, and how to make continuous oversight operational while the licensed verification sector is still developing.

Source: unite.ai

Share this article:

More from Olivia Hart

Olivia Hart

Show full bio

Staff writer covering people, launches and day-to-day practice in entrepreneurship, startups and venture funding at Business Bearings.

2,378 articles

Related articles

« Previous articleNext article »