We’re living through an AI-fueled “Black October” and things are changing fast. Hackers are using AI to breach countless companies. AI tools have uncovered security flaws that could leave 70 percent of all websites vulnerable. Governments are scrambling to secure their sites. Banks, too. And because cyberhacks can hurt supply chains, some longtime AI watchers are going full prepper, stocking up on food, cash, and more.
In the center of this story is Anthropic, one of the world’s leading AI companies, which was explicitly founded on the promise that it would be a “good guy” company in a world of dubious tech giants.
Anthropic says all the latest developments are very serious. Its security chief wrote last week that things are “going to get a lot worse before they get better.” Two weeks ago the company warned that AI models could soon self-replicate on the internet-connected devices in your home, like smart fridges and robot vacuums. And its co-founder now says “the singularity” — when AI systems speed up technical progress in a way that’s hard for humans to comprehend — “is here.”
You might think, given its view of the risks, that Anthropic would be pumping the brakes on its cutting-edge AI progress. Just the opposite. Anthropic filed for a blockbuster initial public offering on Monday, chasing a valuation of $2 trillion even as it warns in the IPO filing that AI could “pose catastrophic or existential risks to humanity.” Every day, it is racing toward recursive self-improvement (RSI), where AI models are smart enough to build the next, even smarter AI models by themselves.
How does Anthropic justify that?
As its CEO, Dario Amodei, reminded the world last week, he built the company on one big bet: that instead of driving a race to the bottom, where companies skimp on safety in their rush to beat competitors in the market, Anthropic could actually catalyze a “race to the top.”
Why I wrote this story
As I’ve watched Anthropic double down on its “race to the top” strategy, I’ve been uncomfortably reminded of another very risky gambit: Sam Bankman-Fried’s crypto fraud of 2022. There are important differences between the two; this is not a perfect parallel. But both involve a tiny handful of figures, strongly influenced by effective altruism, making huge bets on behalf of many, many other people.
EA culture prides itself on offering a counter to commonsense morality. The community has shown a willingness to take big risks and justify them with complex calculations. Sometimes that pays off. But believing you are intelligent enough to adopt a counter-intuitive, galaxy-brained strategy and end up saving the world with it often involves some hubris. And hubris makes me nervous in a field as high-stakes as AI.
The idea is that if Anthropic proves it can build AI that’s both powerful and safe, the market will expect both — so other AI companies will then feel compelled to compete on safety, too. That may sound nice in theory, but it means Anthropic has to keep building ever-more-powerful AIs.
“The forces that drive the race to the top are hugely more effective if the company practicing them is at the frontier,” Amodei wrote. “And looking at that strategy as a whole, I’m proud of our track record and stand by it.”
In other words, Amodei is sticking to his bet.
But Amodei is gambling with, by his and his company’s own telling, the future of humanity. If AI goes wrong, it could lead to our gradual disempowerment, a galling concentration of power, or even complete extinction.
So there’s a very basic question that the public deserves a good answer to right at this pivotal moment: What is the evidence that Anthropic’s stated strategy — compete at the frontier so as to drive a “race to the top” — is actually successful overall?
From the outside, it looks like Anthropic has done more, on balance, to drive a race to the bottom. (Disclosure: Future Perfect is funded in part by the BEMC Foundation, whose major funder was also an early investor in Anthropic; they don’t have any editorial input into our content.)
In fact, part of the problem may be that Anthropic continues to treat this as a classic arms race. Specifically, the company seems to model it as a prisoner’s dilemma: If Anthropic stopped building more powerful AI, another player — whether it’s China, OpenAI, or another company — would just build it anyway. In this kind of dilemma, if the other player is going to defect, it’s rational for you to defect, even though nobody actually wants that outcome.
But what if this framing misses the point?
Game theory can make a huge gamble seem sophisticated, while making the obvious alternative — just don’t build a doom machine — seem naive. Yet the fact that Anthropic and others are now racing toward RSI makes it much more likely that we could lose control of AI and start to see massive damage to our world. When the cost of defecting from safety is that you create a serious risk of catastrophic harm — or of all humanity, including you, going extinct — is that still a “rational” move?
Instead of telling yourself that you can galaxy-brain your way through this with fancy enough game theory, maybe you’re better off just thinking of this as something much simpler: Russian roulette.
More and more leading voices in AI are starting to recognize that.
“We should stop AI progress now, starting unilaterally with Anthropic and/or OpenAI stopping — and then broadening out as a second step to more AI companies and negotiations with China,” Geoffrey Irving, who was until recently the chief scientist at the UK AI Security Institute, told me. “Unilateral pauses can’t hold for long, but they are the right first step: When you’re doing a very dangerous thing, the correct action is stop, even if others are also doing dangerous things.”
Anthropic did not respond to a request for comment.
The case that Anthropic has done more harm than good
Let’s be clear: The question is not whether Anthropic has done anything good on the safety front. It has. Standing up to the Pentagon, temporarily holding back access to frontier models, and publicly advocating for safer AI policies are a few examples, among others.
But the real question is whether the company’s “race to the top” strategy is working on balance — whether it’s net-negative or net-positive for safety.
Nathan Calvin, the chief strategy officer at AI safety advocacy organization Encode, recently set off a firestorm among tech watchers, writing on X, “When I look at many notable actions that Anthropic has taken over the past year, that evidence seems most consistent to me with the view that Anthropic is primarily trying to win the race to superintelligence, and that other considerations (including safety) are fundamentally secondary.” Here’s his list of what Anthropic has done:
And this is all just in the past year. If we rewind the tape further back, we see Anthropic pushing back on a landmark bill to regulate AI. Rewind even further back, and we see indications that Anthropic’s core strategy may have been flawed from the start. According to Reuters, in late 2022, OpenAI CEO Sam Altman heard rumors that Anthropic was building an AI-powered chatbot. So he ordered his employees to ship an OpenAI chatbot first. That chatbot was called ChatGPT. We all know the rest.
The rivalry between OpenAI and Anthropic seems not only to have hastened the arrival of generative AI, but also to keep pushing release dates forward to this day. The two companies have been known to release new or cheaper models within hours of each other, each one keen to protect its market share.
Even if you think Anthropic’s behavior was justified in 2022 because publicly demonstrating the risks of AI also demonstrates the need to regulate AI — a common claim within the company and AI discourse writ large — you have to ask: Is it still justifiable now, when AI has reached this level of power and sophistication?
Anthropic has indicated that it’s getting closer and closer to achieving RSI. In January, Amodei himself wrote, “I am concerned that LLMs are approaching (or may already have reached) the knowledge needed to create and release [biological weapons] end-to-end, and that their potential for destruction is very high. Some biological agents could cause millions of deaths.” And he wrote in September, “Given the accelerating rate of AI capability development, it’s my worry that in 6-12 months such a swarm could be capable of taking over the entire internet with a persistent botnet.”
But at this point, Amodei and his staffers are themselves perhaps the primary ones accelerating the rate of AI capability development.
“Anthropic seems to be doing the whole RSI thing more than anyone else, other than perhaps OpenAI,” Calvin told me. “And I just think that seems insane given their publicly stated views on the risks.”
“I think people at Anthropic should reckon more with the fact that it seems like the modal outcome is that the largest impact that Anthropic will have had is dramatically accelerating the race to RSI,” he continued. “On the current trajectory, everything else they will have done and thought and ideals they had will be a rounding error compared to their contribution there.”
Anthropic should be willing to invite scrutiny on this point — and, if it believes it’s actually making the field safer than it otherwise would be, to show us the receipts. How? It could put before Congress a bill that actually strengthens safety, instead of weakening it. It could stick to its former Responsible Scaling Policy and get other companies to do the same, instead of jettisoning its commitments. It could influence other labs not to build RSI, instead of leading the charge.
In the absence of such evidence, the “race to the top” logic just doesn’t hold water.
What if Anthropic just…stopped?
Amodei has begun advocating for “pacing the frontier” of AI development. “We must slow the pace at which we improve the capabilities of AI models,” he wrote in September.
The catch is that Amodei’s plan to slow down is contingent on others agreeing to slow down too — other AI labs, to start with, and also other countries. But more experts are now saying that it would be better if a frontier company like Anthropic unilaterally stopped.
“I’m glad that he proposed that we pace, but like, now we have to actually do it — and it seems like he’s saying he’s not going to do it unilaterally,” Daniel Kokotajlo, a former researcher at OpenAI who became a prominent whistleblower, told me. “That’s not very helpful, Dario! You should lead. What’s all this about ‘racing to the top’ and ‘leading by example’ if you’re just waiting?”
Voluntary unilateral action by individual companies is of course no substitute for federal regulation and international treaties. But Irving argues that it’s the right first step, because if the companies wait for governments to rein them in, it may well be too late. Governments are notoriously slow. An individual company can act today.
There’s a perennial objection here: “But China!”
Yet the idea that we’re in a classic arms race with China has always been flawed.
For starters, AI is not just one thing with one purpose, like the atomic bomb. It’s a much more general-purpose technology, like electricity. So what matters isn’t simply who gets it first, but who uses it in which ways. Besides, if the harms from AI are massive, then those harms are relevant for the other party, too. The Chinese don’t want to see catastrophic harms to all of human civilization, either. So we shouldn’t write off the possibility of coordination.
Former Vox reporter Kelsey Piper recently wrote about a colloquium at Anthropic where one participant objected: A decade ago, Facebook considered acquiring the tech that would later power TikTok. But, facing pressure due to privacy and safety worries, the company passed. That didn’t end up protecting us from the scourges of a world with TikTok — all it did was hand China an early monopoly on the tech. Piper writes:
Was this a policy success? Did we want to repeat it?
We were obliged to grudgingly concede that it didn’t sound like a policy success. We did not want to repeat it. China might be no excuse for the U.S. labs to do reckless things, but it is a real consideration.
Our next step might be bilateral talks with China, then. But by the point where you agree that Anthropic’s plans might depend on bilateral talks with China, you’ve already conceded most of the steps of Anthropic’s worldview that seemed so much like madness 10 minutes ago: continue working on AI for now, try to negotiate a slowdown, but don’t abandon work at the frontier and all the advantages that come with it.
But TikTok is not a good analogy for RSI. The former may be a cesspool that rots our brains and amplifies ethnic hatred, but its dangers don’t compare to the dangers of RSI. The TikTok case may indeed be best approached as a classic prisoner’s dilemma. But with RSI, the potential downside is so catastrophic that anyone who proposes to save the world precisely by building the tech that endangers it is making a hell of a gamble on behalf of everyone else. At the very least, the burden of proof should be on them to prove their confidence is justified.
AI insiders often think those outside of their industry can’t possibly understand the technical, geopolitical, and moral complexities they’re facing. But commonsense morality is commonsense for a reason: It tracks intuitions that are usually right (such as, if you don’t want a doom machine, maybe don’t build a doom machine) and trains us to be appropriately skeptical of huge, counterintuitive bets to the contrary.
Precisely because the AI landscape is a complex system — there are so many actors (including different AI companies and different governments), and actions from each can trigger actions from others in ways that are hard to predict — believing you’re smart enough to take highly counterintuitive actions and end up saving the world as a result is an extremely risky gambit. Instead, maybe it’s best to just do the straightforwardly right thing: Stop.
To be clear, “stopping” doesn’t have to mean peeling back everything that’s been done in AI so far. The powerful AI systems that are already out there can’t be put back in the box. And we may need some AI to defend from the AI that’s already out there.
“I’m not saying shut down all of your data centers,” Calvin said. “But I’m saying that in particular, pushing the frontier on RSI and making it so that you’re going to be the company that is doing more fully autonomous development than anyone else in the world, does that actually make any fucking sense?”
Increasingly, both AI insiders and the public are shouting no.
Some insiders, perhaps aware that humanity is perfectly capable of shutting down overly dangerous tech development (we’ve done it before), are speaking out and quitting jobs at AI companies. At public gatherings, like the recent Pro-Human Assembly in Washington, DC, and the People’s AI Assembly in New York City, people are piling on the pressure to rethink the AI race. States are drawing up their own legislation to rein in the tech. And a single X post from Calvin got so much attention that no less than Amodei himself felt compelled to respond. The AI companies are being forced to pay attention.
Attention may not be all you need, but it’s a start.
