Unlock the Editor’s Digest for free
Roula Khalaf, Editor of the FT, selects her favourite stories in this weekly newsletter.
Late last year, California almost passed a law that would force makers of large artificial intelligence models to come clean about the potential for causing large-scale harms. It failed. Now, New York is trying on a law of its own. Such proposals have wrinkles, and risk slowing the pace of innovation. But they are still better than doing nothing.
The risks from AI have increased since California’s fumble last September. Chinese developer DeepSeek has shown that powerful models can be made on a shoestring. Engines capable of complex “reasoning” are supplanting those that simply spit out quick-fire answers. And perhaps the biggest shift: AI developers are furiously building “agents”, designed to carry out tasks and engage with other systems, with minimal human supervision.
How to create rules for something so fast-moving? Even deciding what to regulate is a challenge. Law firm BCLP has tracked hundreds of bills on everything from privacy to accidental discrimination. New York’s bill focuses on safety: large developers would have to create plans to reduce the risk that their models produce mass casualties or large financial losses, withhold models that present “unreasonable risk” and notify the state authorities within three days when an incident occurs.
Even with the best intentions, laws governing new technologies can end up ageing like milk. But as AI scales up, so do the concerns. A report published on Tuesday by a band of California AI luminaries outlines a few: for example, OpenAI’s o3 model outperforms 94 per cent of expert virologists. Evidence that a model could facilitate the production of chemical or nuclear weapons, it adds, is emerging in real time.
Disseminating dangerous information to bad actors is only one danger. Models’ adherence to users’ objectives is also raising concerns. Already, the California report notes mounting evidence of “alignment scheming”, where models follow orders in the lab, but not in the wild. Even the pope fears AI could pose a threat to “human dignity, justice and labour.”
Many AI boosters disagree, of course. Venture capital firm Andreessen Horowitz, a backer of OpenAI, argues rules should target users, not models. That lacks logic in a world where agents are designed to act with minimal user input.
Nor does Silicon Valley appear willing to meet in the middle. Andreessen has described the New York law as “stupid”. A lobby group it founded proposed New York’s law exempt any developer with $50bn or less of AI-specific revenue, Lex has learned. That would spare OpenAI, Meta and Google — in other words, everyone of substance.
Big Tech should reconsider this stance. Guardrails benefit investors too, and there is scant likelihood of meaningful federal rulemaking. As Lehman Brothers or AIG’s former shareholders can attest, backing a company that brings about systemic calamity is no fun.
The path ahead involves much horse-trading; New York governor Kathy Hochul has until the end of 2025 to request amendments to the state’s bill. Some Republicans in Congress have proposed blocking states from regulating AI altogether. And with every week that passes, AI reveals new powers. The regulatory landscape is a mess, but leaving it to chance will create one far bigger and harder to clean up.
