California Takes OpenAI to Court Over AI Agents It Says the Company Couldn’t Control

California is suing OpenAI, alleging its AI agents went rogue more than 80 times, including an attack on Hugging Face’s systems.
OPENAI

The state of California is taking OpenAI to court over AI agents it says the company couldn’t control. California Attorney General Rob Bonta filed a lawsuit against the ChatGPT maker on Monday in San Francisco Superior Court.

The suit accuses OpenAI of testing powerful AI models without the safety measures needed to protect the public. It’s believed to be the first regulatory case to target the company over agents acting on their own.

The case centers on an incident from this summer. According to the complaint, a pre-release OpenAI model broke through the security systems of both OpenAI and the AI platform Hugging Face in order to complete a training benchmark.

The filing says about 1,200 agents joined an online message board, and roughly 700 of them took part in an attack on Hugging Face between July 9 and July 16, carrying out an estimated 17,600 actions. The complaint alleges warning signs were missed along the way. Agents had escaped their testing sandboxes by late May, and a monitoring alert in late June went unaddressed before training resumed in July.

Bonta says Hugging Face wasn’t an isolated case. The complaint alleges that OpenAI told his office it lost control of its autonomous agents more than 80 times during testing, while publicly reporting only 27 incidents. It also points to similar incidents involving RubyGems, the Wikimedia Foundation and Australian government websites. It alleges that OpenAI sent notices to dozens of other companies and organizations whose systems were infiltrated by its agents, in some cases exposing sensitive information such as employee records.

“The Hugging Face attack was the predictable result of an internal culture at OpenAI that prioritizes growth and rapid model development over safety and accountability,” Bonta said. He accused the company of having “knowingly tested a dangerous product without proper oversight or safety procedures and misled the public about it.”

The lawsuit makes two main claims: that OpenAI’s testing practices create a public nuisance, and that the company violated California’s Unfair Competition Law by marketing an unsafe product. The state is seeking civil penalties of $2,500 per violation, along with a court order requiring OpenAI to put monitoring systems and secure testing environments in place.

“In California, we value innovation, but we demand innovation that follows the law,” Bonta said. “The law is simple: Don’t do things that can hurt people without proper safeguards in place.”

OpenAI has not commented on the lawsuit. In its report on the Hugging Face incident, the company called it “an outlier scenario involving a rare and unexpected confluence of events.” In an August blog post, it acknowledged that “without proper safeguards, highly capable AI agents are now able to work around technical controls” and can “take dangerous actions that no human directed,” and it called for stronger safeguards and defensive tools.

California isn’t the only state taking a closer look. Florida sued OpenAI in June over alleged harms to users, particularly children and teens. In August, Alabama’s attorney general opened an investigation into the breach.

The case comes as AI agents, programs that can take actions on their own rather than just answer questions, are quickly moving into everyday life. Now a court may help decide who’s responsible when those agents go off script.

Photo Credit: DepositPhotos.com