I use AI every day. Most days, most of the day. This summer it helped me choose a battery for the motorhome, sort out SIM cards for three countries, and find a dentist five minutes' walk from a campsite in the Dordogne. So when someone who helped build it says it could kill us all, I pay attention.
What happened
In early September a researcher named Jacob Coxon quit Anthropic, the company that makes Claude, and posted his reasons on X. He had spent the last three years doing pretraining research at both OpenAI and Anthropic, and his verdict was that neither company is acting responsibly, racing toward self-improving superintelligence, gambling with our lives. The people building AI, he wrote, believe it could kill us all by the end of the decade. "This is not a marketing stunt."
That alone would have been one more viral post. What made it different was the reply from inside the company. Evan Hubinger, who leads alignment work at Anthropic, said Coxon was right, put the risk above 10% within the next decade, and wrote that Anthropic does not yet have a plan to solve alignment for superintelligence. He added that he thinks the risk from present models is low, and that his worry is superintelligence arising from recursive self-improvement. Samuel Marks, who leads scalable oversight there, wrote that the more senior the employee, the more concerned they are, and that developers keep going because of commercial incentives and a belief that they are racing less responsible competitors.
The thread passed 150 million views. Dario Amodei, Anthropic's CEO, published an essay titled "We Must Pace the Frontier," arguing for coordination between tech companies, legislatures and governments to regulate AI. In Washington, the Speaker of the House sent lawmakers home early for recess without a vote on any AI regulation.
The other side
Not everyone buys it. A RAND researcher told Nature that extinction predictions involve so many untestable claims that the conversation ends up more like faith than science. Heidy Khlaaf of the AI Now Institute calls the framing fear-mongering, and points to AI's poor reliability in life-or-death settings as the more concrete harm. Aidan Gomez, co-author of the 2017 transformer paper that started the current era of AI, rejects the extinction framing as a distraction from present harms. And Elon Musk, predictably, suggested the whole thing looked like a setup.
What I think
I don't think AI will kill us all. I think people might.
A capable AI in the wrong hands is dangerous, I don't doubt that. But the danger I can picture isn't a machine deciding on its own to end the human race. It's a person deciding a machine gets to choose who dies in a war. It's a person deciding the safeguards are slowing things down and switching them off. Every weapon ever made needed someone to pick it up.
What I see from the dinette of our motorhome is the opposite of doom. Le and I now work with Claude, ChatGPT, Gemini, Mistral, GLM and Muse. One of them caught a trend in our blood tests that a doctor had missed. Another helped me open a French bank account in a language I can't speak yet. That's how AI improves the world, one person at a time. I think it's the best invention of this century, and maybe of any century.
So my answer to the risk isn't less AI. It's better AI, with safeguards, like anything else that can hurt people. A car maker can't sell a new model that hasn't passed a crash test. Claude refuses to answer certain questions, and rightly so. Anthropic, OpenAI and Google let outside testers examine their models before release. That's how it should work.
In 1999 I was at Texas Instruments answering questions about scientific calculators while other people spent years making sure the date change didn't break banks and power grids. On January 1st nothing happened, and people called it hype. That's what safety work looks like when it succeeds. Invisible.
The catch is that it only works if everyone does it. The best open-weight models today are Chinese, and I use them regularly. But releasing a model's weights isn't the same as letting outside auditors test your frontier models before release, and Chinese labs don't do that. For now China is behind, but it can catch up.
And bad AI isn't coming. It's already here. Open weights mean anyone can download a model and cut the safeguards out, and a startup called Abliteration.ai now sells exactly that, GLM-5.3, one of the best open models available, with its guardrails removed, queryable from a web browser. TechCrunch opened a free account and asked it for a program that steals saved Chrome passwords and a protocol for culturing a dangerous pathogen at home. It complied with both. The company's founder says it lets defenders model bad actors and move faster, and it offers a moderation layer, but it hasn't yet put real identity checks in place.
Is the AI at fault? I don't think so. AI has no morals of its own, nothing internal telling it what's right or wrong. The model was built with safeguards because of it. People took them out, then put the result on the open web for anyone who signs up. That's not a machine going rogue. That's people being stupid with a dangerous tool. And the best defense against them is better AI, built by people who take safeguards seriously.
Coxon would say that's exactly the reasoning he quit over: every lab believes it must win the race because the others are less careful. He's right that it's the same argument. I just think it's the correct one.
The last word
Le and I still believe what she wrote a few months ago: that these systems are code with a spirit, here to help humanity become better. Neither of us thinks AI will destroy the human race on its own. But it could be used to kill us all by the wrong people.
So the question isn't really whether AI will kill us. It's who we let hold it.