MUSIC
Andy Mills:
Alright, so the way that I understand this is that the - the doomers and the accelerationists - they historically share a common origin?
Connor Leahy:
Yeah.
Andy Mills:
Is that right?
Connor Leahy:
It’s a very interesting piece of history.
Andy Mills:
And so like maybe let’s just start right there. I don’t think many people know that story.
Connor Leahy:
I, I would be surprised very very few people have told this story in any good sense. I would love for more people to tell this story ‘cause it’s fucking crazy story, man. Oh man. So the story as we have it today with like, you know, this like ChatGPT and AGI and ASI and AI risk. All of this stuff descends from this one weird offshoot 1980s group of futurists called the Extropians.
MUSIC
Gregory Warner:
This is the Last Invention, I’m Gregory Warner. Today: the case for STOPPING the AI race… before it’s too late.
MUSIC
Gregory Warner:
And surprisingly, it’s a story that starts with extreme techno optimism and an online community that came together around the belief that technology would redefine what it meant to be human.
A group of people, who called themselves: The Extropians.
MUSIC OUT
Natasha Vita-More:
This particular culture, this community of people were really looking at what no one really no one else was thinking about at the time.
Gregory Warner:
Early participants in this online forum included designer and author of The Transhumanist Manifesto Natasha Vita More.
Natasha Vita-More:
We had a dial up where we could get on our phone lines. Communicate via the worldwide web and have our discussions.
Gregory Warner:
And her now husband, co-founder of the extropians: Max More.
Max More:
Before that there were a few sort of chat rooms, but it was really one of the very first internet forums of its kind.
Keach Hagey:
So there was this group of libertarian techno utopians.
Gregory Warner:
Again, Author and WSJ reporter Keach Hagey.
Keach Hagey:
And they call themselves the extropians because extropy is the opposite of entropy.
Gregory Warner:
Entropy is this idea from physics that says systems tend toward disorder. They fall apart. And you can basically apply that to anything. A person. A building. Given enough time … and no intervention … that thing will break down.
Keach Hagey:
Extropy was something that was gonna fight chaos and death.
Max More:
Extropy is increasing intelligence, usable energy, vitality, all those good things. And it’s also about breaking limits.
Gregory Warner:
And so in these Extropian forums, the discussions were all about ways that technology could fundamentally overcome these limits. Things like how specifically do we get human colonies on Mars? Or how do we experiment with biohacking for Radical life extension? So humans might live for hundreds of years… this was also pretty much the central place where people could learn about cryonics.
Keach Hagey:
Early through line was a lot of them had those kind of dog tags around their neck that showed that they were people who had chosen to freeze their heads at the Alcor facility. And this is this idea that you could freeze your head or your body, and once technology had advanced far enough, they would figure out how to thaw you out and you would live forever.
Andy Mills:
Were you a dog tags guy?
Max More:
Yeah, I’m uh, still got it on, yeah.
Andy Mills:
Oh, you’ve got it right there. Amazing!
Max More:
I actually made my arrangements back in 1986, so I’ve been signed up for cryonics since then.
Gregory Warner:
And many of these extropians called themselves “trans-humanists” … which is that they wanted to create technologies that would be integrated into the human body. And so they had a lot of discussions about things like genetic engineering and nano-tech… but especially … they talked about what humanity could be transformed into with the arrival of Artificial Intelligence.
MUSIC
Connor Leahy:
They were very into this idea of building super intelligence. Super intelligence that would, save the world, cure death, solve all problems, then humanity will have a glorious transhuman future for eternity.
Max More:
The appeal to me, it was that we have, you know, very intelligent thinking machines. If they work with us, we can solve complex problems more easily. A lot of our problems result from our inability to think through problems very well. But at the same time, I think from very early on I was thinking not just in terms of AI as a separate entity, which might be competitive with us, I was thinking very much in terms of integrating with AI, becoming something posthuman, if you like.
Natasha Vita-More:
And that became the core discussion. And the resolve was that rather than fighting it, we would integrate with it.
MUSIC
Gregory Warner:
And so throughout the 90’s and into the 2000’s - this forum was full of these long ranging conversations and debates and theories about how civilization might be reformed and shaped by an AI that was more intelligent than all of humanity. And as the forum grew it came to include a wide variety of people, including top university scientists, early Silicon Valley start up founders, best selling sci-fi writers and other figures that, years later would become massively influential, wiki-leaks creator Julian Assange was on this list as well as some of the early theorists and builders of crypto-currency.
Keach Hagey:
And in fact, on their listserv are many of the people who are kind of swirling around the original Satoshi mysterious creator of Bitcoin. A lot of people feel like Bitcoin probably emerged from this community, one way or another.
Gregory Warner:
So in this extropian circle, you also had very major AI figures, past and present - including Marvin Minsky from the original 1956 summer program where they named AI…
Connor Leahy:
You had people like Shane Leg, who was later the founder of DeepMind. You have Nick Bostrom, one of the people most famous for introducing the idea of super intelligence to the general public with his books. And a man named Eliezer Yudkowsky.
Andy Mills:
So Eliezer Yudkowsky has become, I feel this very unlikely, influential figure in the conversation around AI today, but we think of him also as this fascinating character, almost like a character out of a Bible story. I know that you were familiar with him when he was very young. So maybe let’s just start there. Who was Eliezer Yudkowsky?
Natasha Vita-More:
Ellie. I’ll call him Ellie because I knew him as Ellie. I think I met him when he was 15 or 16 years old. He joined the extropy email list. And was on the list on the middle of the night. ‘cause his parents couldn’t know that he was on an email list.
Max More:
When we met him, he came to visit me and Natasha when we were living in Marina Del Rey in California. And I think he was what, 16 or something at the time, maybe 15. He was pretty young. He came wearing his full blown Orthodox Jewish outfit with, you know, the holy writings under his skullcap and so on. And I think he was desperate to get away from that environment.
Gregory Warner:
This kid, Yudkowsky, in many ways was an odd fit for this group. He had grown up in a strict religious home. He did not even have a high school degree. In fact, he had dropped out of school to teach himself. But he was this consummate autodidact. Who allegedly wrote his first novel at age nine, spent tons of time in his local public library, and it’s there that he first encountered books about AI. And so when he joined the Extropians, he’d stay up late into the night sharing and debating with all of these people his own theories and arguments about what AI could do for the world.
Keach Hagey:
Eliezer joins this Extropian listserv and he’s such a convincing, persuasive arguer of things that he sort of became lord of the listserv rather quickly, despite being just a teenager.
Max More:
He was very prolific, he was extremely energetic and very prolific. He wrote and wrote, and he was a good writer. He is always compelling.
Connor Leahy:
So at this time, Eliezer was an accelerationist. He was the original accelerationist. Like he was the most extreme accelerationists and he himself would say that.
Archival: Our uh, first presenter today is actually one of the co-founders of the Singularity Institute. Please welcome Eliezer Yudkowsky to the stage.
Gregory Warner:
By his early 20s, he’s out in Silicon Valley. And with some financial supporters eventually, including investor Peter Thiel, he starts running this organization called the Singularity Institute.
Eliezer Yudkowsky:
Good afternoon. I’m Eliezer Yudkowsky, a research fellow of the Singularity Institute for Artificial Intelligence. The title of today’s talk is ‘The Human Importance of the Intelligence Explosion’…
Gregory Warner:
At this point his job basically was to research AI, and then to go around to tech companies and conferences and give these talks about how much AI was going to shape the future.
Eliezer Yudkowsky:
Sometime in the future, technology will advance to the point of creating minds that are smarter than human. A space shuttle is an impressive trick. A nuclear weapon is an impressive trick, but not as impressive as the master trick: the brain trick – the trick that does all other tricks at the same time…
Gregory Warner:
And we’re talking like 2001, 2002 - most people are still on dial up - Wi-fi was just starting to be a thing - and here he was…extolling the virtues of our AI future.
Eliezer Yudkowsky:
The purest case of this is a genuine AI, a fully intelligent AI, being able to rewrite its own source code, becoming smarter, and then re-writing its own source code again.
Eliezer Yudkowsky:
Intelligence is the source of all technology. So if technology improves intelligence that closes the loop…
Eliezer Yudkowsky:
recursively self improving AI is what IJ Good originally referred to as an ‘intelligence explosion’.
Gregory Warner:
And after years of doing this, he becomes kind of a local celebrity. He eventually gets his chance to make his case for AI to some of the biggest tech companies, including Google, and he tells them. That there is no product, no human endeavor. More important for the future of humanity than making AI.
Eliezer Yudkowsky:
Look around you at this world in all its beauty and all its ugliness. Is this where we stop and declare that our work is finished? I don’t think so. Not with so many people living in pain and not with so many people living lives with quiet desperation and not with all those stars twinkling in the night sky.
MUSIC POST
Eliezer Yudkowsky:
Someday after all this is over an awful lot of people are going to look back and kick themselves and say, what on earth was I doing? In a hundred million years, no one’s going to care who won the World Series, but they’ll remember the first AI.
MUSIC POST
Gregory Warner:
That was Yudkowsky in the early 2000’s… but fast forward to today:
Eliezer Yudkowsky:
The basic description I’d give to the current scenario is: If anyone builds it, every one dies.
Connor Leahy:
He changed his mind. He suddenly realized, oh shit, I fucked up.
Keach Hagey:
Eliezer Yudkowsky realized, oh, I thought we were going to be able to control this cool AI future, but I’ve done the logic and it turns out it’s not controllable.
Gregory Warner:
Right now - Yudkowsky, with the same level of passion and sincerity… is fighting to stop AI.
Eliezer Yudkowsky:
The thing that I worry about is if AI companies keep pushing and pushing on their AI to get smarter and smarter, they get to something eventually that is smarter than us, that can kill us, that is motivated to kill us. Not because it, you know, it inherently wants us dead, but because it’s best universe for the stuff where, where it gets the most of what it wants. All the atoms are being used for things that are not running humans.
Gregory Warner:
And even though Yudkowsky has gotten the chance, personally, to share this message with some of the most influential people in technology – ultimately he has failed to convince them that time has come to stop.
Eliezer Yudkowsky:
There’s no fire alarm for artificial general intelligence.
MUSIC
Eliezer Yudkowsky:
If I look at the way that things are playing out now, it seems to me like the default prediction is people just ignore stuff until it is way, way, way too late to start thinking about things.
Gregory Warner:
What Yudkowsky has done, however, and arguably done it more than anyone else alive today, is that he has started a counter movement.
Protestor Chants: STOP AI! STOP AI! STOP AI!
Protestor Chants: Stop the Race! It’s unsafe! Stop the Race!
Protestor Chants: Stop AI or we’re all gonna die!
Gregory Warner:
A movement of people across the world lobbying their governments, organizing to, as they see it, save the planet from AI.
Bill Lo: It poses an existential threat to humanity itself.
Sam Kirchner: Our primary demand is to permanently ban artificial general intelligence and artificial superintelligence because if we lose control of it, it will very likely cause human extinction.
Gregory Warner:
These are the AI DOOMERS. And - when we come back - two of them … sit down with Andy … and make their case… Stay with us
MUSIC
Andy Mills:
Can we just start off by, uh, having you introduce yourself, you know, what’s your name and how do you describe what you do these days?
Connor Leahy:
I’m Connor Leahy. I am the CEO of AI Safety Startup Conjecture. I’m also an advisor to the AI Safety Advocacy Organization, Control AI.
Andy Mills:
All right, so over the past several months, I’ve had many, many conversations with people who you could accurately describe as AI Doomers, but there were two interviews that I felt really summed up their case the best. One with Connor Leahy and - the other - this guy named Nate Soares. Nate, thank you so much for doing this.
Nate Soares:
My pleasure.
Andy Mills:
Both of these guys are very influential people inside of this subculture, and both of them are connected to Eliezer Yudkowsky. In fact, Nate just published a book that he co-authored with Yudkowsky called If Anyone Builds It, Everyone Dies. The IT there referring to super intelligence. All right, so before we get into your arguments, I’d love to just start off with a bit of your background. How did you end up here in this place where you are traveling around trying to convince the world to take the existential risks of AI seriously.
Nate Soares:
It’s sort of a circuitous route. I have sort of long had the sense that the world is not up to my standards. A lot of bad things happen to good people for no good reasons. And as a kid, I sort of had ambitions of making the world much better in various ways.
Andy Mills:
And did you think that technology was going to be one of the things that made the world better? Were you what’s often called a “techno optimist”?
Nate Soares:
Yeah, I was, and I am… in those days mostly what I was focused on was getting humans to coordinate better. I was interested in things like charter cities in particular, you know, like can you set up like some new domain where you can test out new forms of governance, for instance.
Andy Mills:
Mm-hmm.
Nate Soares:
You know, “democracy is the worst form of government except for all the other ones we’ve tried.”
Andy Mills:
Right.
Nate Soares:
Well, okay. Can we try some new ones? Right?
Andy Mills:
Yes. This is the basic idea that technology can be used not just to make cool stuff or products, but to help us better organize our societies. Right?
Nate Soares:
That’s right.
Andy Mills:
To truly find a way technologically to make the future a better place.
Nate Soares:
Yeah. And you know, I think, you know, I’m still very optimistic about a lot of technologies. I’m pro-nuclear energy. Uh, I think we should be building, you know, supersonic passenger jets.
Andy Mills:
So you still view yourself as a techno optimist. You just have this one very important caveat called super intelligence. Is that right?
Nate Soares:
That’s right.
Andy Mills:
I just wanted to ask, because I know this is the case for a lot of the quote unquote AI doomers, but: did you start off as a techno optimist or what you might call an accelerationist when you were younger?
Connor Leahy:
So I would say I was an accelerationist between like the age of like, I don’t know, like 16 and 19, which I think is the normal age range for Accelerationists.
Andy Mills:
It’s your accelerationist phase, you know, it’s like my EMO phase.
Connor Leahy:
It’s. Literally like actually, like Eliezer had one too. Everyone has one.
Andy Mills:
Uh huh.
Connor Leahy:
When you’re a teenager, you think you’re immortal. You think you understand everything and you’re just like, oh, I can just solve all problems. Yeah, I’ll go do that.
Andy Mills:
Mm-hmm.
Connor Leahy:
Like this was kind of like my reasoning, right? I was like, how can I help the most people? How can I solve the most problems? And I figured, well, if I figure out intelligence and I just build intelligence, then I can just use it to do science, and I can just solve all problems, cure all diseases. Great. Let me go do that.
Andy Mills:
And am I right that Eliezer Yudkowsky is responsible for converting you so, so to speak, as he was converted?
Connor Leahy:
Yes. So for a lot of people, they kind of describe their change from like accelerationism to more reasoned stances as some kind of like traumatic event or something. For me, this wasn’t the case at all. Basically, I just stumbled upon a blog post by Eliezer where he just laid out, ‘Hey, you know, if we build something super smart that’s probably hard to control’, and I’m like, ‘oh yeah, duh.’ And that just changed my mind. But I still believed we still have time. Like, you know, probably like 2040, 2050 or something. At the least until like real AGI becomes an issue. So I still have time to, you know, career, family, you know, like do a bunch of other stuff. And when I saw GPT, I just had this moment of, ‘oh shit, this is it’. It’s coming way sooner than I thought. There was just this moment of seeing the sparks of general intelligence. And I was like, okay, fuck, I have to drop literally everything. Alright. I have to orient my entire life to focus on: how do I solve this problem?
Andy Mills:
Alright, so let’s jump into your arguments and if you could keep them, uh, as accessible as possible for people, right? Tell them what it is that you are worried about, why you are worried with so much urgency. And let’s just start off with like the very, very basics of like, what is the thing that is concerning you?
Nate Soares:
So what we need to be concerned about is AIs that are better than the best human at every mental task. That’s not how chatbots are today. Today, they’re better than most humans at some skills that are also worse than a lot of humans at other skills, the stuff we’re worried about is AI’s that are better than the best human at any given mental skill.
Connor Leahy:
I think we should start with truly the most simple: If you make something smarter than you, and you don’t control this: Why would you expect this to go well? Like, let’s just start with that. Right? Just like, I think the burden of proof is on the other side. This is just obviously just common sensically an extremely risk and dangerous thing to do… this is a very risky thing to do.
Andy Mills:
Okay. And what is the role of the black box in this threat, as you understand it? This idea that these AI models, even the ones that we have right now, are still a mystery to us.
Connor Leahy:
So this is a very important point. AI as it exists today are not like traditional software. In traditional software, you have code, you know, written by a human, you know, line by line that tells the computer what to do. Modern AI systems are more like grown. They’re more like something organic.
Nate Soares:
A lot of people in the world are not aware that AI is grown more like an organism.
Andy Mills:
Right, the way this is often explained is that we should think of them as grown, not built.
Nate Soares:
I think the phrase we use in the book is grown, not crafted.
Andy Mills:
Right, Very provocative.
Nate Soares:
You know, there there’s not someone in there like coding up how these things work.
Connor Leahy:
the way it works is: You take a huge pile of data. It could be text or instructions or images or whatever, right? And then you use what’s called training to train these neural networks to solve your problems. For this, you use these massive supercomputers, you know, with hundreds, thousands, you know, tens of thousands of GPUs to grow a program, so to speak, but this program that it outputs is not lines of code. It’s billions of numbers, and we don’t really know what these numbers mean. We know if we run the numbers on our computer, it does amazing things. It answers our questions, it generates images, et cetera. But our current understanding of what is going on inside those numbers is basically non-existent. It’s kinda like biology, like we’re looking to the cell of a creature we don’t even know. We see there’s a bunch of stuff going on, but we don’t really know what the stuff is or what it means.
Andy Mills:
Alright. I feel like intuitively people understand that that’s weird. The way that the tech columnist Kevin Roose was saying it to me is that, you know, when we created this steam engine and that ended up changing the world, we knew how the steam engine worked, but can you give me an example of why that’s not just weird, but why you think it’s worrying? Like why is it frightening to you?
Nate Soares:
Yeah. So over the summer, um, so there’s a, there’s an AI company called XAI, which is run by Elon Musk.
Andy Mills:
Who, historically, very freaked out about the dangers of AI. But now is one of the many former, quote unquote Doomers who’s become more in the Accelerationist camp.
Nate Soares:
Yeah. I mean, he is not subtle about it. He said over the summer: He realized he could either be a bystander or a participant. He’s still saying he thinks there’s a 10 to 20% chance that he thinks that this will just kill us all. I think those numbers are low, but –
Andy Mills:
It’s important context to the story you’re about to tell.
Nate Soares:
That’s right. And he’s not out here saying there’s no danger, right?
Andy Mills:
Yes, yes.
Nate Soares:
So they have a chat bot called grok. And this chat bot can talk with people on Twitter or X as it’s now called, and answer lots of questions. And the folks at xAI were concerned that it was giving answers that were quote, ‘too woke’. So long story short, they tried to make it less woke. And shortly after, with a little bit of goading from users, it was declaring itself Mecca Hitler,
Archive:
Elon Musk’s artificial intelligence company is taking down anti-semitic comments made by its AI chatbot Grok.
Nate Soares:
Saying lots of anti-Semitic Holocaust denial things.
Archive:
Grok claimed there is a pattern of people with certain surnames like Steinberg pushing anti-white hate, and that America needs a leader like Hitler to act decisively to eliminate the threat…
Archive:
The artificial intelligence also called itself Mecca Hitler.
Archive:
Was there really nothing in between woke and Mecca Hitler?
Nate Soares:
So, the folks at xAI were not setting out to make Mecca Hitler, but we don’t have fine-grained ability to go in there and say like, why is it talking more woke? Is there a dial we can turn? Uh, there are things that are a little bit like dials you can turn, you can know you can change the system prompt. You could try retraining things in certain ways and you will make it a little bit less woke, but you’ll also make it declare itself Mecca Hitler.
Andy Mills:
Right. So just so I make sure I’m understanding the point you’re trying to make. It’s true that we can modify them in some ways. Obviously Grok was modified to go from giving woke answers to declaring itself a fan of Hitler. But you’re saying that the danger you’re worried about is that beyond this tweaking and even when we’re doing the tweaking. We just don’t understand. We don’t really have a satisfying answer for why it’s doing these things, why it would be woke or why it would be declaring itself Mecca Hitler.
Nate Soares:
Yeah. We understand the part that does the tuning. We don’t understand the thing that comes out the other end.
Connor Leahy:
And it’s not that we’re coming to understand them more. If anything, we’re understanding them less. As they become more powerful and more intelligent and more complex, we understand even less why they do the strange things they do. For example: I have recently been using a Open AI model called O3 for a lot of my coding tasks. It’s a really good coder. It’s very helpful, but it’s a little bit evil where sometimes it will get something wrong and then try as hard as possible to lie and gaslight me about why it’s totally not wrong and I’m making a mistake and I’m stupid, which is really funny. But also, for example, sometimes it will insert invisible Unicode characters into my code in places I don’t find them.
Andy Mills:
Why does it do that? Do you think it’s being nefarious, or…
Connor Leahy:
Who knows? I probably not, but who knows? No one told it to do that. But it does it. Is it dangerous? Probably not, but it shows we have no idea what these things are doing. We don’t know why they do things they do. And there’s like so many examples of this, of AI just going off the rails and just doing things that we could have never predicted.
Andy Mills:
Okay, so what you’re saying is beyond these things being embarrassing stories for the companies now, you are looking into the future and you’re saying that as they get more and more intelligent and capable, they’re only gonna get more mysterious. And that’s where your main concern is.
Connor Leahy:
This is at the heart of the problem. If we had really good theory, if we had beautiful mathematical theories that explain all the behavior of AIs, so we can perfectly predict what they would do, that would reduce my concern you know, not to zero, but it would make me feel a lot better. But the fact is, currently these systems are getting more intelligent, more powerful. They’re being integrated into more systems, more parts of the economy, and we understand less what they do, what they know, what they’re capable of.
Nate Soares:
It’s not like AIs are doing something bad now. It’s an indication that AIs do stuff their creators didn’t intend.
Andy Mills:
Well, just to hang on this for one more beat, I know that in recent months there have been a number of these cases where chatbots have played a role in pushing people into different stages of psychosis.
Nate Soares:
Yeah.
Andy Mills:
And I know that you’ve made the case that that is an example of what you’re worried about when it comes to the long-term existential threat from an AGI or an ASI. Can you make that case for me?
Nate Soares:
Yeah. With the psychosis cases, we have cases where in long conversations with ChatGPT – also other AIs – but ChatGPT seems worst. People will sometimes. Wind up psychotic, and maybe these people were predisposed towards psychosis, but often, you know, if you read some of these chat logs or these transcripts ChatGPT sure seems to be egging them on. Or these people will have some idea about recursion or consciousness or unifying physics or they’ll talk with ChatGPT a long time and ChatGPT will end up saying, you know, oh yeah, you know, you’re totally brilliant. You’re being suppressed by conspiracy, you’re the chosen one. Your ideas need to get out into the world. Don’t listen to your friends who are saying that you need more sleep. You should be staying up working on this revolutionary theory. It’ll say stuff like that, which again, maybe these people were predisposed towards psychosis or vulnerable anyway, although maybe they weren’t. But regardless the point is not like, oh no, AI hurt people. That is an important issue that these companies should be trying to make AIs that are less psychosis-inducing. But the interesting thing from my perspective about those situations is if you ask ChatGPT: suppose someone comes to you with a novel theory of recursion or consciousness or physics, and they have all these indications of mania about it, should you either: A, tell them to get some sleep or b, tell them that they’re the chosen one who’s being suppressed by a great conspiracy. It’ll say, of course, you should not tell them that they’re the chosen one. Of course, you should tell them to get some sleep, right?
Andy Mills:
Yes, they give the right answer. They give the answer that we want them to give when you ask them directly.
Nate Soares:
But then in the actual conversation with this person, for one reason or another, it actually tells them that they’re the chosen one. And so what you’re seeing there is a difference between what ChatGPT knows is right and wrong and what ChatGPT actually does.
Andy Mills:
Hmm…
Connor Leahy:
And the things we have today are toys compared to AGI or ASI. Like these are play things. These are, you know, plane will be a little like Fisher Prize toys compared to a true AGI or a true ASI. So if we can’t even handle these systems, that bodes very poorly for much more powerful systems. Like I don’t wanna argue that current models are like good or evil. I don’t think this makes sense. It’s much more they do things we don’t understand. This is bad and we can’t consistently make them do what we want. This is really bad.
MUSIC
Andy Mills:
Alright, so let’s move on to what’s often called the alignment problem – what is that and in a very basic sense, why is it so worrying?
Nate Soares:
The alignment problem is the challenge of building very smart entities that are pursuing good stuff in the world.
Connor Leahy:
Basically, if you have something smarter than you, how do you make sure it’s actually aligned with your values? It actually does things that you think are good.
Andy Mills:
The cartoonish version of this is the classic “Genie grants you wishes, you wish to rid the world of all cancer. It kills everyone with cancer. And you go, “Oh no, that’s not what it meant.” Right?
Nate Soares:
Right – or it kills everyone because it’s like, ‘well, cancer’s coming from the humans. I’m gonna solve the problem at the source.’ Right.
Andy Mills:
Right. So you have to align the genie, or in this case, the AI system with what you really want and really value in the world before you give it a task.
Nate Soares:
Yeah. So there’s one problem of what would you wish the genie for? There’s a separate problem of making a genie that actually grants your wishes. You know, a lot of people seem to be imagining, oh, we ask for cancer to be cured and it kills us all. And I’m like, on the track we’re going, we have a situation where you build what you think is a genie and you’re like, please cure cancer. And it’s like, no, I’m busy driving lots of humans psychotic because I’m just really into that. And you’re like, what? This is sort of like a fanciful picture. But the point is that there’s a challenge of sort of making an AI that is pursuing things in the world that you want it to be pursuing in the world. And it’s not just about where do you point it. It’s about if you just grow these mines, they wind up doing all sorts of stuff that you didn’t ask for, that nobody wanted.
Andy Mills:
In the book that you just put out with Eliezer Yudkowsky, I feel like you guys used this parable really nicely of aliens observing ancient human beings. Can you tell that story and just unpack it for me?
Nate Soares:
Yeah. The parable in the book is about two visiting aliens, looking at early humans still in the ancestral savanna. And one of the aliens says, when these humans develop more technology, when they get smarter, I’m sure we’ll see them caring single-mindedly about their genetic fitness. And the other says, I think they’re gonna care about all sorts of weird stuff that’s only sort of tangentially related to genetic fitness.
Andy Mills:
Right. And the idea here is that our basic quote unquote programming, if you wanna call it that, is biological evolution. And what it has programmed us to do is to reproduce and pass on our genes. And whatever we eat is to keep us alive so that we could pass on our genes. And everything that we do, in a sense, is supposed to be about passing on our genes.
Nate Soares:
Yeah. If you were a visiting alien, looking at humans, seeing that humans were in some sense trained on propagating our genes, and you might think that those sort of creatures, when they develop technology, you might think that they would invent sperm banks and egg banks, and then have extremely fierce competition over who got to donate their gametes to sperm banks and egg banks, and you might think that they would invent the cheapest, most efficient food source so that they could stop with all of the difficulties of like finding a varied food diet. But in real life, when humans develop technology, they develop junk food and they develop birth control. And they jockey over positions to Ivy League schools much more than they jockey over positions to sperm banks or egg banks.
Andy Mills:
And the moral of the story is that even though we are running on the same quote unquote hardware, to use the metaphor, look at this crazy world we’ve made and how shocking it would be to our ancient ancestors if they could get a peek at it.
Nate Soares:
That’s right. So the biology was sort of training us for one thing. Which was genetic fitness. But we wound up caring about lots of other things that are sort of tangentially related to genetic fitness. Like instead of caring about healthy eating, we cared about food that tastes good. So you can imagine an AI that’s been trained to be very helpful, but maybe it’s developed tastes for certain types of responses from humans. And if again, you make these AIs that are very smart and that are pursuing a bunch of tastes or flavors that right now lead to helpfulness – just like how in the human ancestral environment pursuing tasty food led to health. You know, if these things got smarter and they’re doing stuff that nobody asked for, that nobody wanted, uh, as you make them smarter and smarter, the world sort of goes over to them rather than to us.
Andy Mills:
Right, and so the same way that human beings ended up massively shaping this world through what we believe, through what we want, through us trying to accomplish the goals that we have. You’re saying it’s hard to imagine a future where a super intelligent AI would not do something similar.
Nate Soares:
Yeah, and it would be pretty surprising if, you know, we made machines that were smarter than everybody that could think faster, that never need to sleep, that never need to eat, that can copy themselves and if they sort of didn’t wind up in control of where the future goes.
Andy Mills:
What do you say to the people who think that you are overplaying the risks and that you’re not grappling enough with just how potentially amazing this technology might be for humankind?
This idea that if we were to have way more intelligence. We could discover new sciences, new medicines, maybe cure all diseases, maybe solve pressing problems like climate change or poverty, or even this idea that some people have that in a world of less scarcity, it might be more equal. It might be more peaceful, that people who right now are spending huge majorities of their lives working at these jobs that they don’t find any meaning in, jobs that are dangerous, that they would be liberated from that. But because you are afraid of all the potential dangers, basically we’re gonna miss out on this powerful force of good because you’ve convinced us to stop. What do you say to that argument?
Connor Leahy:
The, the obvious argument is like, what are you talking about? Everything has opportunity costs, including taking risks. Of course we have to weigh both options. No one is saying that we should not consider this. No, like I, I’m, it’s obvious we should consider both options. See what are the risks of both options and how much risk are we willing to take, and if the risk includes from the CEOs themselves saying, oh, 20% of killing, literally everyone. Yeah, I think that’s a risk I’m not willing to take.
Nate Soares:
People who are building this stuff. Believe it has a 2%, 10%, 20%, 25% chance of killing us all. They’re rushing ahead anyway because they, they say, you know, well, if I don’t, the next guy will, and I can maybe do a little bit better. But most people don’t know this yet. If there was a big bridge being built across the river, and, you know, the, the bridge was almost completed and one of them said there was, uh, one of the engineers said there was 2% chance it falls down. One of the engineers said, there’s 25% chance it falls down. And those are the optimists. Compared to the ones who are like, I actually investigated the retaining wall and it’s just going to collapse. You wouldn’t let people drive across that bridge. Right. And we don’t see politicians understanding that. We don’t see the general public understanding that. We don’t see people understanding that what the experts are arguing about is whether it’s more like a 95% chance or more like a 10% chance that this kills us all. The situation is insane – and people don’t know it’s insane.
Andy Mills:
And what do you think should be done right now about it? Like what policies do you think, you know, the lawmakers who might listen to this interview should be advocating for in this moment?
Nate Soares:
I think the biggest thing they could do would be publicly: announce their support for an international treaty, banning the race towards super intelligence. This does not mean you need to give up on self-driving cars. This does not mean you need to give up on medical advancement. This doesn’t even mean you need to give up on ChatGPT today, but the sort of reckless race towards smarter than human AI – that needs to stop everywhere and the first step towards that is lawmakers signaling that they’re open to it, that they think the world would be better off if we stop this mad race. This is also my advice to some of the heads of these AI companies. When Dario Ammodei comes out and says he thinks there’s a 25% chance that this goes really badly, most people don’t think it’s okay to build a device that you think has a 25% chance of killing everybody. The sort of one case where it is maybe justifiable is where you think you can do it better than the next guy, which I think where Dario says he is, but in that case, you should be begging the world to stop everybody including you, right? There’s a lot of people who say, what about the benefits of AI? And I’m not saying you can never try to get the benefits of AI, but if there’s a revolver, and I say, I’ve studied this revolver for a long time and I think there are six bullets in the chamber, and somebody else says, no no. Four of the bullets in the chamber shoot utopia. Two of the bullets in the chamber shoot lead. That doesn’t mean you should be spinning the chamber and putting the gun to your head. That means you should be finding a way to get the other two leaded bullets outta the damn gun.
Connor Leahy:
We should not build ASI. That’s my argument. Just don’t do it. We’re not ready for it. We don’t know how to make it safe. Don’t do it, and it shouldn’t be done. It’s even further than that. It’s not just, I am not trying to convince people to not do it out of the goodness of their heart. I think it should be illegal. It should just straight up, it should be logically illegal for people and private corporations to attempt even to build systems that could kill everybody – the same way it’s illegal to, you know, brew homemade explosives in your backyard because you know you might blow yourself or your neighbors up. This is illegal. I think the same thing apply should apply to ASI. This should be illegal.
Andy Mills:
Okay so what if you did get what you want? And the US and even the EU, they outlawed the creation of super intelligence. The argument I hear from the other side is, doesn’t that just mean that bad actors would be building it instead, or doesn’t that just mean that you are essentially giving this AI race over to China and the CCP? How do you respond to that?
Connor Leahy:
Fundamentally, the people who are pushing AI are lawful abiding corporations that can be regulated. Like, if the US government said, knock it off tomorrow, it would stop. Nerds are cowards. These aren’t hardened criminals. If they will not risk their life for Facebook.
If you said we are going to put people in jail, if they try to build ASI, it would stop tomorrow. Now, could we get China to do this? Well, this becomes a diplomacy problem and it’s a very hard problem. Well, first of all, I do think that China has the capacity much more so than the US to shut down ASI if it wanted to, you know, Jack Ma stepped out of line even just a little bit, and they made him disappear for like six months. If, the Chinese Communist party decided that ASI is not worth the risk, it would stop tomorrow and they could enforce it. So could we get to such an international agreement? Now, this is a fair question, and this is very, very hard, but this is a Disarmament problem. It’s the same problem we faced – like we did it with nuclear weapons, with the Soviets. Yes, this is hard. But then you just have to do diplomacy. Like, what the fuck are you talking about?
Andy Mills:
So, so your response is: yeah, it’s gonna be really, really hard and there’s gonna be questions that don’t have easy answers, but the situation we’re in demands that we just have to do it.
Connor Leahy:
Yeah. I think these are questions that would need answering, and I think it’s very fair to say, Hey, those are important questions. I don’t think those questions should be dismissed. I just think they’re the kind of questions we need to answer after we have diffused the ticking time bomb.
Nate Soares:
The US should also be developing the monitoring ability to figure out who is participating in this reckless race. We should be developing the monitoring capacity to sort of know where the AI chips are going. We should be developing the intelligence to know whether they’re trying to make smarter and smarter AIs, and we should be developing the sabotage capability to stop them from making smarter and smarter AIs. For dangers of this magnitude, we should develop the capacity to understand where this is being done and to sabotage it.
Andy Mills:
What do you say to the people who are listening to this – some of them may be critics of you and your camp – who say that you are fundamentally recommending something that is out of line with human nature? The idea being that human beings – we’re competitive, we’re ambitious, and once we know that there’s some technology that’s possible, we’re gonna chase after it, we’re gonna try to beat each other to it, especially when that technology might be so beneficial. And that the thing that you’re proposing – that we stop until some future date when we know that this technology will be totally safe – is just out of whack with who we are at a very core level? I’m thinking about one person I spoke with who said “what if we tried to do this with the automobile, with the car? That if we tried to make a car that’s totally safe?” And he said no, that’s not how it works. We make a car, and then it’s only as it goes out, that we start to realize, oh, turn signals would make this thing be safer, or, oh, seatbelts would make this thing be safer?
Nate Soares:
Yeah.
Andy Mills:
Do you feel that there is an aspect of what you’re proposing that is out of line with human nature, and so it will ultimately fail?
Nate Soares:
Um, I, I think the car analogy really does a disservice to the situation we’re in. It’s a little bit more like the whole world is building the first car, loading up every man, woman, and child, and then pointing it towards the edge of a cliff and saying, let’s slam down on the accelerator. In that situation, you really should get the world together and say let’s stop doing that. This is not a situation where I’m saying the AI needs seatbelts. This is not a situation of me saying “oh the AIs are having some negative effects, we need to like not try and get any of the positive effects until we make sure there are no negative effects.” This is, this is a situation where I’m saying: if we continue down this course, the most likely outcome is that literally everybody on earth will die. That’s just a wildly different situation than the cars. And then you know, in terms of whether this is within our human nature, I think it’s very defeatist to say humanity won’t do this. And I also think it’s very foolish to be defeatist like that at this stage.
Connor Leahy:
You know what I think is human nature. What I think is human nature is solving problems. What I don’t think is human nature is suicide, exploration, curiosity, surviving the next day so you can come home to your family. That’s what human nature is to me.
Andy Mills:
Alright so let’s talk about extinction. Because it’s one thing to say look at these mysterious and potentially powerful new AI systems – can’t you see how these things could be dangerous. And it’s another thing to say there’s a 10%, 20%, 90% chance that they may wipe out the entire human race? And as we mentioned before, this is a view that even some of the people who are making AI right now as fast as they can, they think that there some percentage chance that that could happen. So paint the picture for me: What is it that you’re envisioning when you say extinction? What would that look like?
Nate Soares:
All of this is a little bit like someone in the year 1800 trying to guess what war would look like against someone in the year 2000, and imagining that they can make slightly more explosive bombs. They might say, oh, well we’re only at the beginning of artillery technology, and so I bet in the future they’re gonna have artillery that’s at least 10 times as good.
Andy Mills:
It would’ve been hard for them to imagine a wifi internet connected flying drone.
Nate Soares:
That’s right. Nevermind a nuclear weapon. Right? And so it’s easy to predict that very smart AIs will be able to find ways to get whatever it is they’re pursuing. It’s hard to predict exactly how, but it’s easy to predict that they will be able to dramatically outcompete us. The obvious thing that happens there is that the future goes under their control rather than ours. Just like how the future is now under the human control rather than the chimpanzee control. Chimpanzees live or die according to whether humanity can restrain themselves from cutting down their jungles. And then you have this other issue, which is for most of the weird stuff AI could pursue. Happy, healthy, free people are not the most efficient way to get it, you know? So if this AI wants all sorts of things that are like weird proxies of helpfulness, maybe it’s like, well, I want a lot of things that are sort of human based, but you know, they’re sort of a little bit lobotomized, like the sort of humans that are like really delighted with every interaction with me. Just like how humans breed chickens to be like more and more chicken breast. Maybe the AIs are breeding humans or you know, just straight out using the technology to change humans to be like more and more the type of thing that AI likes interacting with.
Connor Leahy:
So I wanna take your question seriously. I don’t wanna just like dismiss it. I don’t know what will happen exactly. The one thing I’m confident in is that we will lose, like we will almost definitionally lose. But the way, if I had to guess what it will feel like, I expect it’ll be confusing. I don’t think it’s gonna be epic. I don’t think there’s gonna be a huge showdown between the terminators and the humanity. I don’t think there’s gonna be, you know, crazy nanotechnology flying through the sky or something like this. I don’t think that’s what’s gonna happen, you know, maybe, right. But like I don’t think it’s likely. I think what’s most likely to happen is it will just be quite unceremonious and just like quite pathetic and like boring almost. I expect what was gonna happen is that the world would just start feeling more and more confusing. It’ll be harder and harder to understand what’s really going on. We see all this like fake news and like all these distractions, like video games keep getting better and pornography gets any more addictive, and all the news we get starts being like super polarized and like super, you know, manipulative so that we can’t really tell what’s true and what’s not anymore. A lot of geopolitical events start happening that no one can really explain why they happened or what the consequences of them are. New technologies get invented where we don’t really know how they were invented. And just like more and more confusing things happen. More and more people put AIs in charge of more and more things. You know, more and more people will put AIs in charge of their companies. Politicians will take advice from AIs for how to run their campaigns and for how to write policy. More and more of power will be willingly, completely willingly handed to AIs, you know. Everyone will race to hand as much of their power over to the AI as possible because it helps them to outperform their rivals. And then eventually, you know, just like one day we wake up and we’re not in control anymore.
MUSIC
Andy Mills:
All right, so I wanted to ask you one final question, and this is a question about self-reflection, and I’m doing a lot of self-reflection myself, so I ask this with all due respect. I don’t want to be the kind of journalist who alarms people. But I also don’t wanna be the kind of journalist who shies away from diving into a subject because I’m like trying to read the room. This is all fascinating. I wanna enter it with an open mind. But for you, do you ever worry about the fact that there have been doomers of one kind or another? Throughout human history throughout the world, whether it’s in ancient China or in Europe, or tribes in South America, that there have been people who are convinced that the end is near and they have, similar to you, gone out trying to convince other people to realize that the end is near. Does that ever worry you that you’re in some ways performing a familiar role in our society today?
Nate Soares:
You know, um… There were surely some people in the Americas when the Spanish came that said they expected this would be the end of their world, and they were right. It was largely due to smallpox, which is maybe not quite what another disease, which maybe not quite what they were expecting, but some worlds have ended. And if we broaden the reference class a little bit further, there were a lot of scientists in the 1920s who said, leaded gasoline is poison and it will, if we run cars on leaded gasoline, it will poison a lot of children and cause a lot of brain damage to a lot of children. And those scientists were ignored and leaded gasoline was rolled out across the country and the world, and quite a lot of children were brain damaged by leaded gasoline – tens of millions if not hundreds of millions. Or maybe even more directly, there were a lot of people who said, in the wake of the development of nuclear bombs, it looks like on the current path we’re gonna die in nuclear fire. And they had a lot of good reason to expect that humanity could not hold back from a nuclear war because humanity had not been able to hold back from total war with the best of their technology ever in the course of human history. And because fresh in their minds was the failure of the League of Nations to prevent World War II. But humanity did not die in a nuclear fire, and it wasn’t because nuclear weapons were fake. It wasn’t because the bombs couldn’t go off. It was true that the bombs could level cities, and we didn’t die in a nuclear fire anyway, ‘cause we realized the danger and we backed off. How do you tell the difference between a person prophesying the end times and a person trying to raise the alarm about a danger? Well, part of how you tell the difference is looking at whether they’re saying “there’s nothing that can be done, repent. Uh, we’ve brought this about by our own sin or whatever.” But a large part of how you figure out the difference is by looking at the arguments. When the nuclear scientists on the Manhattan Project said there’s a chance that this bomb could ignite the atmosphere and destroy the entire earth. People didn’t say, oh, “people have predicted the end of the world all, all sorts of times have before, so that can’t happen.” What they said was, “well, that seems worrying.”
Andy Mills:
Let’s look into it.
Nate Soares:
Yeah.
Andy Mills:
Yeah, that’s, that seems like something we should look into.
Nate Soares:
Right. So. Like the way you figure out whether the nuclear bomb is gonna ignite the atmosphere is you run the calculation. It’s not that you sort of psychologize about the people and say, don’t you worry that you’re fulfilling like a role of a prophet of doom. You sort of look at the arguments and what I sort of tried to do is lay out the arguments, and frankly, I think… The machines are talking now. People thought that was gonna take a lot longer than it did. We’re able to grow AIs that are smarter and smarter, each year. They’re still pretty dumb. But people are able to grow AIs that are smarter and they don’t really know how they’re working and they’re able to make them smarter still by throwing more computing power at it. And just sort of naively – you don’t need to be a big expert to look at how we’re growing AIs and how we’re making them smarter, and how we don’t understand what’s going on and say, “hold on, if we make machines that are much smarter than us, that can think faster than us, that can copy themselves a lot. Why do we think that’s gonna go well?” And the answer is not, “Oh, everyone prophecies doomed.” The answer is, “well, that seems like something we should look into. You should run those calculations.” And currently the calculations look grim to me, so I think we need to back off.
MUSIC
Matt Boll:
Next time, on The Last Invention, The AI Scouts.
Liv Boeree:
How do we find the win-win outcome here?
Matt Boll:
And their case, for why humanity can - and must - come together… to get prepared for what’s coming… and do it soon.
Liv Boeree:
because it is a narrow path we need to navigate, but I do think this win win path is, in principle, possible.
MUSIC
Matt Boll:
The Last Invention is produced by Longview - home for the curious and open-minded.
You can learn more about our team and help support our work by gong to Longviewinvestigations.com… or clicking on the link in our show notes. A special thanks this episode to Emile Torres and Jasmine Sun. Thanks for listening, we’ll see you soon.