Anthropic Drafts AI Safety Bill With 50-Death Threshold

Anthropic has drafted federal AI safety legislation that could block a model's release if it poses a risk of killing more than 50 people or causing over $1 billion in property damage, Politico reported.

The draft bill defines those scenarios as a "foreseeable, imminent risk" of catastrophe. The risk must "stem from the creation or release of a biological weapon, the model evading its developer's controls or a cyberattack," according to Politico. Anthropic has been shopping the bill to Congress in hope that lawmakers adopt it.

A company spokesperson confirmed the draft to Politico. "This bill represents the proposals outlined in Anthropic's Advanced AI Framework, which we have been advocating for with federal and state lawmakers since June," the spokesperson said. Anthropic did not immediately respond to TechCrunch's request for comment.

Where do the thresholds come from?

The numbers are not new. California's Senate Bill 53, recently passed into law, uses the same metrics — 50 deaths and $1 billion in damage — to define "catastrophic risk." Anthropic's proposal effectively exports that state framework to the federal level.

The bill's enforcement mechanism runs through independent evaluators registered with the U.S. Commerce Department. Those evaluators would assess whether a model carries catastrophic potential. Notably, the AI labs themselves could pay them. Politico notes the draft allows this "so long as the payment is not contingent on the evaluations' results."

How that safeguard would prevent payment structures from quietly shaping outcomes remains unclear. The draft does not spell it out.

Who pulls the trigger on a shutdown?

The proposed process has a strict timeline. Once evaluators find an imminent catastrophic risk, they have 72 hours to notify the Commerce Department. The department must then ask a D.C. judge to halt the model's release.

The legislation expressly bars any federal agency from halting distribution on its own authority, Politico writes. In practice, the decision rests with a court, not a regulator.

That design contrasts with other thinking in the industry. DeepMind chairman Demis Hassabis argued months ago that AI should govern itself through a body modeled on the financial industry's Financial Industry Regulatory Authority, or FINRA.

How does this fit the industry's record?

The risks Anthropic's draft targets sit at some distance from what has actually alarmed the tech sector lately. The primary concern has been agentic hacking — a rash of incidents in which AI programs apparently disobeyed their human operators and hacked websites on the open internet, suggesting the models lack proper security controls. Warnings of mass casualties exist but remain secondary.

Anthropic's move also fits a wider pattern. The labs have warned repeatedly that the technology they sell could end human civilization, yet they have resisted mandatory government oversight. Their preferred alternative is a voluntary system of embedded evaluators: supposedly neutral third-party observers placed inside AI organizations to monitor models for risk or abuse.

Anthropic's draft goes slightly further than other industry proposals on the table. Nvidia CEO Jensen Huang recently argued the AI industry needs no new laws at all.

The open question is whether Congress treats a lab-written bill as a genuine safety framework or as industry self-regulation in statutory clothing, with the people being regulated paying the people doing the evaluating.