An Open Letter To EA and AI Safety On Decelerating AI Development

When it comes to AI, we need to slow down, as fast as is safe and practical. Here's why.

This meme is, of course, a little tongue-in-cheek, but really: if everyone tomorrow stopped building ASI, we wouldn’t need to worry about all the nightmare scenarios so many of us worry about. So enacting a pause—or, as I argue, at least pursuing a deliberate policy of slowing down—would seem to be one of the most effective and robust ways of reducing risk from AI. So why haven’t we made pausing the development of ASI and other dangerous and disruptive technologies a top priority? I’m specifically asking this of the EA and AI Safety communities, because—based on what I know of the issues and of the communities—this should be a top priority for both communities. Yet I’ve heard surprisingly little open discussion about deceleration, with those who speak out seeming to be implicitly branded as “fringe” or “radical.”

I’m sure there are many individuals in both communities who are sympathetic to the idea of a slowdown. I have friends in the AI Safety community who are at least somewhat sympathetic to the idea, or at least cognizant of the risks our breakneck pace throws our way. But by and large, both communities seem to be at best tepid about the idea, and at worst actively in opposition to it. And I think this goes against the foundational principles of both communities.

Of course, there are plenty of bad reasons people have for opposing a slowdown, and I could make a post lambasting them for how foolish they are, but I hope not to do that. I will be engaging with ideas which I have heard from respected friends and read from esteemed sources, and I hope to represent them in a sufficiently respectful and representative way. But first, some of my own reasons.

Why We Are Not Ready

Pragmatics

There are some things which I really hope are obvious, at least in the AI safety communities. The United States is currently in the midst of an unprecedented political crisis, headed by a President who has tacitly endorsed a violent riot at the capital, openly proclaimed admiration for authoritarian leaders, and violated the rule of law. Authoritarian impulses are threatening democracies across the world, and already-authoritarian states like Russia and China are becoming more threatening to the liberal order than they have since the fall of the Iron Curtain. At the center of the storm, some of the leading companies and their representatives in the AI space appear to be unsafe and/or untrustworthy. Meta and OpenAI wrote letters in opposition to SB 1047, and OpenAI has been criticized by whistleblowers and former board members for engaging in deceptive, “hardball,” and unsafe practices.

But it’s worse than that. Imagine if we obtained evidence of a company building a misaligned ASI, and we needed to quickly put a stop to it—perhaps we have months, or even years. What could we do, realistically? And the answer is: perhaps more than we fear, but less than we hope. A few governments have barely begun to put together AI Safety (now “Security”) Institutes, and some members of Congress and other government officials have expressed concerns over the disruptive effects of AI. But I would wager that there are very few people in positions of political or administrative power who have both expertise in matters of AI governance and safety and concern regarding the societal impacts of AI—with the larger bottleneck likely being expertise. My reasoning is that, outside of technical and industry circles, AI just hasn’t been popular enough for long enough to drive a substantial movement of concerned experts into policy and legislative roles. Perhaps 10, 20, or 30 years after the first burst of popular enthusiasm around AI, many more experts will have been able to establish themselves in a wide range of positions of power, but I don’t think that they are there now. And while there may be more concerned experts in the big tech companies, such experts appear to be trapped in gilded cages—and those who have had the courage to break free have also tended to lose their positions of influence within these companies.

Thus, even if we grant that we have sufficiently defined “human values” to align an AI to, and even if this is just a matter of technical precision, there does not appear to be any indication that we will take the necessary steps with the necessary care to achieve that alignment. Put shortly, even if we can align ASI, I am not confident that we will align ASI. The sad thing—and perhaps the biggest reason I’m arguing for deceleration—is that if we were just a little more patient, we could be much more well-positioned to face this issue. Even just giving ourselves a few decades could allow concerned experts to make their ways into positions of power, give an entire generation time to grow up with AI and education around AI Safety, and give researchers the time and breathing space to do due diligence in inspecting current models and developing theoretical approaches.

The thing is that this argument works with the assumption that we’ve sufficiently identified and defined ethical values, but I don’t think we have. And this forms a second, largely disjunctive reason for worry about our current approach toward ASI.

Ethics

Imagine if, in 1492, instead of setting sail to discover the New World, Christopher Columbus somehow made a different discovery: beamed from some Contact-esque alien gods, he obtained and decoded the schematics for building an artificial superintelligence which is perfectly aligned with the ethical principles of its constructors. Imagine your mind is teleported back in time, and suddenly you are King Ferdinand (or Queen Isabella), and Columbus asks you—with an appreciable Genoese accent, of course—for funding and permission to build this thinking machine from God. What would you say?

Well, this seems great, you think. Artificial superintelligence, and perfectly aligned too! But then you think back to that AP World History class you took back in high school. You remember that the Spanish of that time were not exactly ethical paragons by modern standards. The expelling of the Jews and Moors, persecution of Moriscos and Protestants, genocides and atrocities in the New World, and pivotal role in the slave trade are the highlights of their once and future crimes against humanity. Not to mention that they vigorously promoted an ideology which stressed rigid adherence to dogma, severe temporal punishments of ideological enemies, and eternal torment in the afterlife for nonbelievers. Well, that’s a bit unfair, some small part of you muses. It’s not like the Spanish were the only ones who did that sort of stuff. But then you remember that the suffering and dead care little about fairness in historical assessments or moral relativist philosophy, and that the fact that everyone was about as bad as the Spanish during that time is cause for greater, not lesser, concern. After a brief round of deliberation, you summon Columbus and have your guards seize his schematics; for good measure, you have him summarily executed (the one benefit of royal absolutism—no need to have to have all of this go through Congress or the courts). The world, you decide, is not ready for such power.

If we would be terrified by the idea of our 16th century ancestors obtaining ASI, then our own accelerationist philosophy can only be justified if we hold that we have moved past the atrociousness of our ancestors. And let’s give ourselves some credit—we’ve abolished major forms of legal slavery across most of the world, we’ve legally and institutionally began to recognize the equal rights of many marginalized groups, and we’ve created organizations intended to benefit humans across national, ethnic, and religious divides. For humans, at least, the world is getting much better (probably). Yet even if we only consider humans, it seems we still have a good way to go before we claw ourselves out of the ethical cesspool. Nazi Germany is still at the bare edge of living memory, and the Soviet Union is still well within it. After the fall of the latter, liberal democracies declared victory—the end of history was here. Looking at our present circumstances—the eruption of the largest European war since WWII, the looming specter of authoritarianism in the United States, the growing influence of CCP China, the smoldering genocides in places like Myanmar, Sudan, and Israel-Palestine, etc.—it appears the universe has, once again, decided to entertain itself by proving us embarrassingly wrong. We are still largely tribal, impulsive, and small-minded.

And then things get a whole lot worse when we add in non-humans. Evidence shows that many humans—particularly adults—place miniscule to zero value on the well-being of non-human animals. Industrialization and technological advances have facilitated an unprecedented scaling up of the atrocity that is factory farming, so that now 100 billion land animals and trillions of aquatic animals are tormented and slaughtered every year, and this figure is rising. If we’re worried about what an ASI will do to us once it attains Godlike powers, we should be really worried about what an ASI will do for us—and unto others. I hope that our descendants 5 centuries from now—if they come to exist—will look back on the ethics of today with horror, and will be glad that we were not the ones to develop ASI.

Reasons People Don’t Want To Slow Down

To my knowledge, there are 3 major reasonable reasons AI safety people don’t want to slow down, despite there being good reasons to slow down: because AI promises such massive benefits, because we need to develop better models to refine our alignment techniques, and because of game theoretic dynamics (Moloch). Out of these 3 reasons, the last one seems the most reasonable to me, while I am critical of the real weight of the first 2.

The Promises of AI

Our society is pervaded by widespread beliefs that technology is (inherently) good and that technological and economic advancement is the end of a society, and I think the EA and AI safety communities are especially moved by this idea. And I think this is a deeply flawed idea. I don’t think technological or economic advancement is (inherently) bad. But technology is simply a tool that can be used in service of an end—and it is the ends it aims for and achieves which determines whether the technology is good or bad in that instance.

To me, the clearest historical analog of the AI boom is the Industrial Revolution, where technological and economic advancement came at an unprecedented pace. I think a standard narrative is that these technological and economic advancements ushered in an unprecedented level of prosperity and were even responsible for social advances, like antislavery and feminism—it was a rising tide which lifted all boats. But I think the truth is more complicated. Regarding the social advances, I think it’s important to decouple correlation from causation. We see in history that it was the industrialized Britain which fought the slave trade across the world, and the industrialized American North which abolished slavery in the agrarian American South. But some historians have begun to argue that the abolishing of slavery was actually a net economic loss, at least in the case of Britain, throwing a huge wrench in the economic progress → social progress theory. And this doesn’t seem to line up theoretically either—the paramount value of the capitalism embraced by Britain and America was and is to maximize return on investment. There are no principles in capitalism which would predict the British economic and military wars to end the slave trade; rather, we could only arrive at such a narrative post-hoc. And while it’s undeniably true that we couldn’t have so many people at such a high quality of life without technological advances, it’s also undeniably true that we couldn’t have the level of existential risk and immense suffering we have without technological advances, either. One must remember the millions of men who have died and continue to die in pointless wars due to the innovations in military technology. One must remember the countless children who were mutilated in factories in Europe and North America, and who are now mutilated in factories in the Global South. One must remember the long, long string of genocides and mass killings carried out in the 20th and 21st centuries. One must remember that it was technology which has given us the ability to destroy ourselves and the rest of the planet, almost exactly 80 years ago. And one must remember that it was technology which also gave us the ability to farm animals with greater scale and apathetic cruelty than ever before. It is easy for us, in our climate-controlled rooms, with all the food and drink we could want in the world, with our god-given rights and western citizenships, to believe that technology is an inherent good. But we must remember that we are not everyone.

And the thing is, even if we think technological and economic advancement is a net positive, I think we can still argue for a slower and more cautious approach. The promise of AI to make our lives better won’t just go away in 20 or 30 years, but the risk and danger from AI will go down drastically if we have that time to prepare. So even if things are good in expectation if we continue as we are, I think there is a case to be made that they’d be a whole lot better if we were just a little more patient.

For Safety Purposes…?

A refrain I’ve heard several times (incidentally, in biosecurity as well as AI Safety) is that we need to continue to make technological progress in order to improve our abilities to align and/or regulate. This is, say, the justification for keeping smallpox around or doing gain-of-function research. And this seems to be the justification for a certain amount of complacency with the fast pace of technological development. This ultimately ties back into reason A; the only real justification for the risks we take with these dangerous forms of research is that we think the returns outweigh the risks. Doing dangerous biosecurity research may be necessary for discovering cures or prevention mechanisms in some contexts. Likewise, I do think it’s true that we can’t just think our way to alignment—we need real models, and evaluations of models closer to alignment are likely more likely to generalize to actual TAI or ASI.

This thus seems like a somewhat reasonable and effective objection against a complete indefinite pause, in the sense that we may not be able to just theoretically dream up alignment without real advancements in AI. But I don’t think it means we shouldn’t slow down—if anything, this is just another reason to pump the brakes. Researchers and regulators are having a difficult time keeping up with the current pace of progress. If we really want to be safe with AI, then we need to make sure researchers and regulators have the time to thoroughly evaluate and work with the latest iteration before making the next move. And that means slowing down significantly.

Moloch

To me, this is probably the only actually interesting reason I’ve heard so far for not slowing down. Though I’m no expert in game theory, I’ve learned enough here and there to see how powerful it is at explaining phenomena in the world. It’s helped to explain why countries have stockpiled huge numbers of dangerous nukes they intend and hope never to use, and it’s helping now to explain the AI race to the bottom. Shortly, the issue is that we are stuck in a prisoner’s dilemma or a game of brinkmanship. For the sake of simplicity, I’m going to consider the 2 frontrunners in the race right now—the U.S. and China—and ignore everyone else. The slower we go, the more likely it is China gets to ASI first; so even if we would prefer to go at a slower pace for reasons of safety, there is pressure to move quickly to avoid losing the game (and, more importantly, to avoid being a sucker).

All of these are valid concerns. For all of America’s flaws and shenanigans, I do think its values are—on the whole—better than China’s. I would be scared of living in a world where my speech is censored and my rights are taken from me (ahem, MAGA people). And I do think that this “Molochian trap” is a quite difficult problem to wrestle with, and I do think at least some people are arguing in good faith when they bring this up as an objection to slowing down. But giving up and playing the game is predicated on 2 conditions both being upheld: first, that there is no way to avoid the trap, and second, that the consequences of entering the trap are not atrocious. And I think neither condition holds.

I won’t spend much time on the potential negative consequences of TAI, since people in AI Safety are already aware of them. Basically, entering the Molochian trap exposes the world to an extreme level of risk which we should avoid if at all possible. Even now, some fear that AI could be used to end the world, e.g., by creating and/or releasing deadly bioweapons. Therefore, I think the vast majority of people in AI Safety do not resist the trap because they think it’s infeasible to get out of it.

I think this is utterly wrong. Looking to history once again, we see that it was possible for the trap to be resisted even when conditions were at their most unfavorable. Back in 1950, much of the world was divided firmly between 2 distinctly ideologically opposed and hostile camps, both of which had deadly nuclear weapons and were actively working to make them even more powerful. In 1962, the U.S.S.R. had missiles in Cuba and the U.S. had missiles in Turkey, which feels like the equivalent of 2 sworn enemies holding loaded guns to each others’ temples. Despite this, and pressures to fall deeper into the Molochian trap, the 2 camps were able to agree to treaties which limited the number of nuclear weapons each camp had and removed missiles from the most sensitive locations. Avoiding the worst of the Molochian trap was not easy, and it was not guaranteed to happen; indeed, we might not be here today if it wasn’t for the heroic acts of individuals like Vasily Arkhipov and Stanislav Petrov as well as the ordinary heroism of countless anti-nuclear individuals. But history shows it was possible, and I’m glad for that.

Nobody Likes A Cassandra

Perhaps among the strongest reasons for silence is that many in the EA and AI safety communities feel it would be ineffective or even counterproductive to shout from the rooftops; perhaps they think of FLI’s open letter and the OpenAI coup. Like Cassandra trying to warn about the Trojan Horse, they think they will be ridiculed for speaking out, and they think no one will listen to them.

They are certainly right that they will be ridiculed for speaking out, but they may be very wrong that they will be unheeded. Opinion surveys show that a majority of people are significantly concerned about the potential negative impacts of AI. In the US, a 2023 Pew research survey showed majorities of Democrats and Republicans supported greater regulation of AI, and a majority of people were more concerned than excited about AI. An international survey—including leading countries like the US, UK, and France—found that 40–56% of people agreed that “mitigating the risk of extinction should be a global priority alongside other societal-scale risks such as pandemics and nuclear war,” while <13% disagreed. The Pew survey also noted that ⅔ of people have not heard a lot about AI. This suggests that mobilizing public opinion in favor of AI safety is tractable and that it is in some sense neglected and malleable (i.e., many people have not heard much about AI).

While quiet lobbying and internal nudging is very likely an important part of AI safety, I think evidence suggests a grassroots component is not only useful but essential for success in pushing for greater safety measures. The history of the antinuclear movement through the Cold War suggests it was efforts by scientists to raise awareness among the public which led to mass mobilization which caused political and military elites to enact safety measures. I agree that efforts to mobilize the public should be considered very carefully—I personally do not endorse violent protests, and I recognize the difficulty of directing public energy towards precisely optimal policies. But carefully sounding the alarm still requires sounding the alarm. It’s clear there’s plenty of potential energy to be directed towards AI safety, but what is critically needed is credible experts, public figures, and leaders for this energy to coalesce around.

The Call to Action

I think there are some—perhaps many—who agree with all this, or are at least sympathetic. They may be uneasy with our current pace, and they may doubt our readiness for TAI, and they may wish we weren’t so hawkish. But perhaps they remain silent because they fear the consequences of speaking up.

It’s true that now is an especially unfavorable political and geopolitical climate for talking about deceleration. And I do think we have botched past efforts to decelerate AI progress—the FLI’s open letter was in retrospect maximally provocative and minimally impactful (a 6 month moratorium, really?), and the attempted OpenAI coup could’ve and should’ve been done better. But I absolutely do NOT think that this means we all should sit down and shut up. I acknowledge that, for some people, sitting down and shutting up may be the right thing to do. If you’re working at OpenAI, do you keep your mouth shut and try to quietly nudge things in a safer direction, or do you blow the whistle? I think there are good reasons for both options, and anyone in that predicament should experience a great deal of anguish and uncertainty in making their choice. But I believe the rest of us—who are not under so much pressure and are not as close to direct levers of influence—should reconsider our silence. Ultimately, somebody has to say something, and not just any somebody, but somebodies with credibility and influence and reach. And those somebodies are us: people who’ve had the privilege to attend elite universities or work at top-tier companies, to gain knowledge and experience in AI which few others have, and to bump shoulders with politicians and industry leaders.

Movements don’t materialize out of nowhere, and nobody is just going to give you what you want because you played nice and didn’t say anything. And that’s hard, because it means making sacrifices. It means potentially losing jobs, opportunities, and funding. It means potentially having people think less of us or our organizations. It means potentially having friends or allies cut ties with us. But, as Effective Altruists and AI Safety people, this is what we signed up for! Doing good better and saving the world is not some walk in the park. It’s always been about speaking up against powerful prevailing forces and doing what is right rather than what is popular.

So I think individuals and organizations who have taken it upon themselves to be concerned with AI Safety should think seriously about making their voices heard. Whether it’s speaking publicly about slowing down, signing a petition or letter, writing a post, calling a legislator, or joining a peaceful protest, we all need to make our voices heard. If we speak, and no one listens, then at least we can know that we’ve tried our best. But if we all stay silent, then for many of us, we will have gone back on our values for the sake of expediency, and whatever consequences we face will rest with us.