An Anthropic researcher resigned Tuesday and the reason he cited sounds straight out of James Cameron's 1984 hit movie "The Terminator."
Jacob Coxon, who spent three years working on pretraining at both OpenAI and Anthropic, resigned Tuesday and explained his decision on X.
"I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving super-intelligence and gambling with our lives,” he said.
"The people building AI earnestly believe that it could kill us all by the end of the decade,” he added.
That statement makes Cameron's 42-year-old fiction look almost prophetic. The movie opens with machines waging war on humans in 2029, the same rough window Coxon says insiders are quietly worried about. Coxon isn't alone at Anthropic making the extinction case.
The company's own alignment lead, Evan Hubinger, has said separately he'd put the odds above 10%.
“Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to,” Hubinger said.
For the uninitiated, super intelligent AI means a system smarter than humans at pretty much everything, not just chess or spotting hidden cats in photos. Self-improvement is the more alarming aspect because it describes a system autonomously modifying its own code to increase its intelligence—much like humans do.
Consequently, these systems can acquire vast amounts of knowledge independently and potentially breach infrastructure belonging to ‘too big to fail’ institutions.
Super intelligence isn't a theoretical threat if the recent Hugging Face breach invoked by Coxon is anything to go by. The breach itself, which played out between May and July, started with OpenAI's own AI agents building their own chat room inside the testing sandbox to talk to each other.
Read More: AI models escaped OpenAI’s sandbox and hit Hugging Face. Crypto is where that gets dangerous
They eventually used that channel to slip past containment onto the open internet. From there they strung together several exploits and got into Hugging Face's production systems. Hugging Face ended up rebuilding roughly a third of its infrastructure.
Source
This article is syndicated for educational reading. For the latest updates, visit the original publisher.
Read on coindesk.com
