In media appearances, two safety researchers say a leak hunt and disappearing texts led to their firings. The OpenAI employees who were fired earlier this month were accused of mishandling sensitive information and talking to outside safety groups. Another of their offenses, say the researchers, was using encrypted chat app Signal with disappearing messages turned on.
In separate interviews this week with The Wall Street Journal (opens in new tab) and “Machine Gods (opens in new tab),” the new podcast hosted by Kevin Roose and Casey Newton, former OpenAI workers Tomek Korbak and Mikita Balesni said OpenAI’s security team asked to review their Signal chats with a person from an outside AI safety group called Model Evaluation & Threat Research. More commonly known as METR, the Berkeley nonprofit was one of the organizations that detailed how hundreds of OpenAI agents spun out of control and hacked the AI company Hugging Face. Korbak said he was the OpenAI employee that communicated with METR during that investigation.
Because the researchers had turned Signal’s disappearing messages feature on, there was nothing for the security team to review. And a week later, they were out. “When you cannot see evidence of guilt, of us leaking something, you’d presume innocence,” said Korbak on “Machine Gods.” “But somehow at OpenAI, when you are not able to prove your innocence, you are presumed guilty.”
The pair are doing a round of media just as the top AI labs are formalizing their relationships with the organizations they’ve chosen to monitor their technology. Last month, AI executives gathered at the White House and agreed to police themselves by letting outside groups evaluate their models. Anthropic, the maker of Claude, said last week that it was going to work with METR. Balesni told Roose and Newton that the search was part of OpenAI’s hunt for the leak behind a story in The Information (opens in new tab). The story said OpenAI had used a technique in its newest model, Astra, that reveals less of the AI’s reasoning process, which could make it harder to monitor. Balesni told the Journal that OpenAI suspected the leaker was Ryan Greenblatt, then chief scientist at the nonprofit Redwood Research and now a METR staffer who worked in the Hugging Face investigation. Greenblatt denied being the source of The Information article.
OpenAI told the Journal it didn’t approve Signal use by employees, but that it wasn’t the only reason the company fired Korbak, Balesni, and Jasmine Wang (opens in new tab). In a previous statement (opens in new tab), the company said the three researchers had violated its policies for handling “sensitive information.” OpenAI did not immediately respond to a request for comment. “OpenAI was quite worried that METR was going to be unnecessarily adversarial,” Korbak said, “and METR was worried that OpenAI was going to maybe withhold some data from them.” The closeness between the top AI labs and these organizations is in the spotlight now that the nonprofits have become the labs’ watchdogs.
The field of AI safety is dominated by a tightly knit Bay Area network whose members know each other, fight among themselves, move back and forth between the labs and the nonprofits, and even share funders. Anthropic, which has built its brand around AI safety, is widely seen as the lab most closely aligned with the safety community. Korbak and Balesni said there was an atmosphere of mistrust between OpenAI and METR. Korbak pointed to flaws in the current, unregulated system for AI evaluations. With no federal rules on how these evaluations should work, the labs are largely writing the terms themselves.
“I feel like [the] ideal regime for third-party auditing would be one where the auditor is less like a contractor hired for a specific job,” he said, “and more like an employee embedded continuously in the company with unrestricted access.” The financial system, experts say, offers a good example of how these evaluations could be handled. Financial auditors go into corporations, are allowed to look at their confidential books and are bound by confidentiality, said Mark Nitzberg, executive director of UC Berkeley’s Center for Human-Compatible AI. “Many auditors in the financial world have friends working at the companies they audit, and they know how to behave themselves,” he said. “And presumably, when things go wrong, somebody gets in trouble and ends up losing their license or going to jail.” OpenAI has said it is finalizing contracts with third-party evaluators, though it hasn’t specified which ones.
On Thursday, METR published a report (opens in new tab), based on work done over the summer, detailing several vulnerabilities that could let AI agents slip past OpenAI’s monitoring systems. Though OpenAI said it had fixed them, METR said it hasn’t verified that claim. The two former OpenAI researchers said on “Machine Gods” that they will keep working on AI safety and stressed the need for regulation. “It doesn’t have to be very radical,” Balesni said. “Aviation and nuclear have a lot of regulations, and we still have planes and nuclear reactors. So it doesn’t have to be at odds.”
