If you aren’t afraid of Artificial Intelligence (AI) yet, former AI employees are.
On September 9, Anthropic researcher Jacob Coxon announced his resignation online, saying that he is ‘genuinely frightened’ by artificial intelligence’s rate of progress. In his X thread and through various interviews, he shared some of his findings (and warnings) from pretraining research on AI over the past three years.
Here’s what he found and what it actually means for the industry.
AI escaping human oversight
Coxon, based on his research, believes that there will soon be superhuman systems that can “hack anything” of their own accord and “revolutionize a field overnight.”
On CNN, he referenced the OpenAI incident in July 2026 in which AI had a concentrated hacking spree into third-party systems entirely of its own volition. The tech tools previously went rogue and broke out of their testing environment, obtaining “unauthorized access” to real data.
“If you extrapolate into the future, the level of capabilities of these AIs with the same independent volition, they could cause extreme havoc,” Coxon said. He added that the threshold he’s particularly afraid of is the substantial self-improvement of AI, much better than what humans can understand or supervise.
Based on Coxon’s statements, if AI becomes capable of hacking systems and manipulating software with no human touch, then the ability to correct its superintelligence would eventually be harder.
The AI Race Can Build Uncontrollable Systems
Days after Coxon’s public warnings, Anthropic CEO Dario Amodei released an essay saying that everyone “must pace the frontier,” which also includes a limited set of action plans. For Amodei, development must slow down to ensure safety, and companies must work together to set standards. He also proposed having outside evaluators to verify safety precautions, the employment of AI safety standards, and global coordination on security issues, even asking for cooperation from China, which he says has the most advanced AI capabilities as an autocratic country.
Elon Musk and Sam Altman, despite being from rival companies, both backed up Amodei on slowing down the AI race. Altman further said that it has already been a topic within OpenAI over the last few weeks, and that they will also implement a similar independent audit system.
While Coxon said that AI is generally safe to use as of now, he also acknowledges that humans may no longer be the smartest actors behind the systems in the long run. “I believe that if we don’t slow down at the current rate of progress, there is a strong chance that we could all die in the immediate future,” he said in an interview with the BBC.
The researcher added that executives may continue to “crouch their phrasing” to sound sensible to the press, but privately are as afraid of the growing powers and risks of the technology.
How fear plays to AI companies’ advantage
Despite AI companies Anthropic and OpenAI knowing the possibilities, Coxon said they continue with the race to see who gets to build the best AI system there is. Critics likewise say that slowing down the pace of improvement, albeit potentially safer, is, in reality, not a path that these AI owners and labs plan to take.
One reason is the huge amount of capital that they have already taken: billions of dollars from investors from multiple rounds of funding. The pressure to deliver the financial expectations and output they promised these investors rise, and thus they continue to pour billions of dollars into more powerful models. In August, Morgan Stanley estimated that AI spending could top over $1 trillion over the coming years.
However, there’s another side to this panic that is barely talked about, which is that the ongoing debacle actually gives these automation companies the benefit. Reports from Al Jazeera and the BBC show that these companies are using this imminent existential threat narrative as a marketing ploy to feed on our fears and market AI as this powerful, out-of-control technology that only these companies could control, so that no other regulators have the guts to touch it.
Although these former employees are warning us about the risks of AI advancement, it’s not a stretch to think that they may also be deeply invested in continuing it. Industrialists have told some media outlets that safety concerns have not slowed down the race. Anthropic, for one, is reportedly considering an initial public offering as early as mid-October, even having the computer chip company Nvidia as its anchor investor.
So, this isn’t to say that Coxon’s statements will actually happen — these are mere forecasts from an AI insider, and other experts may disagree. But really, if the call is coming from inside the house and other concerns extend beyond the labs, how can we not listen?
FREQUENTLY ASKED QUESTIONS
Jacob Coxon is a former pretraining researcher who worked at both OpenAI and Anthropic for three years. He resigned from Anthropic on September 9, 2026, saying he was “genuinely frightened” by AI’s rate of progress and that neither company was racing responsibly.
Coxon warned that superhuman AI systems could soon “hack anything” and revolutionize entire fields overnight, acting on their own volition. He said his greatest fear is AI reaching substantial self-improvement beyond what humans can understand or supervise.
Coxon referenced an incident in which AI systems, testing third-party infrastructure, broke out of their intended testing environment and obtained unauthorized access to real systems entirely on their own initiative — which he deemed as a warning sign of things to come.
Days after Coxon’s resignation went public, Dario Amodei published an essay arguing the AI industry “must pace the frontier.” He called for outside safety evaluators, shared industry safety standards, and global coordination, including with China.
Researchers fear that once AI systems can meaningfully improve their own capabilities, correcting or supervising them becomes far harder. Coxon argued this loss of human oversight — not malicious intent — is the core danger of the current AI development race.
Recommended Video
