Join us on this fascinating journey as we sit down with Jaan Tallinn, the founder of Skype and Kazaa, to explore the groundbreaking world of AI and its potential implications for society and businesses. Listen in as we tackle the difference between summoning AI and aliens and discuss how it impacts our ability to control the outcomes of AI development. We also delve into the idea of computational universality, the Church-Turing thesis, and how AI is advancing rapidly due to the need for significant computational resources.
Join us on this fascinating journey as we sit down with Jann Tallinn, the founder of Skype and Kazaa, to explore the groundbreaking world of AI and its potential implications for society and businesses.
Listen in as we tackle the difference between summoning AI and aliens and discuss how it impacts our ability to control the outcomes of AI development. We also delve into the idea of computational universality, the Church-Turing thesis, and how AI is advancing rapidly due to the need for significant computational resources. Additionally, we ponder if aligning AI development more closely with the functioning of our brain could lead to a decrease in the computational power currently required for AI.
The conversation doesn't stop there, though. We venture into an examination of the three components that can potentially increase AI power and how context learning allows a model to modify its behavior according to a given context. The risks associated with AI's black box nature, the difficulty of predicting how AI might act in the future, the public's attitude towards AI, its potential economic implications, and the increasing leverage of technology are all on the table. Jaan shares his insights on these critical topics as we underscore the fact that the number of futures that contain humans is a small target as technology advances.
Lastly, we discuss the potential implications of AI and how it differs from the human brain. Jaan provides intriguing insights into the need for regulation and the potential pitfalls of having one company control the compute. We debate the pros and cons of constraining AI experiments and consider the potential risks of centralization versus existential risks. Don't miss out on this illuminating conversation with Jaan Tallinn as we traverse the captivating world of AI.
Key Quotes:
Time Stamps:
(00:24) - AI's Risks and Opportunities
(13:34) - AI Advancements, Risks, and Regulation
(20:13) - Neural Networks and the Need for Regulation
(25:18) - Centralization Risks Versus Existential Risks
Links:
Learn more about Jaan Tallinn
Learn more about the Future of Life Institute
Connect with Vahe
Check out all things Cognaize
Jaan: [00:00:00] If you have a lot of people with a lot of privileged access, that's actually an indicator of just general broken IT operations. It's showing me that there's other things in the business that aren't right. Therefore, we have to band aid it by having people with deep access that can go manually fix it.
Vahe: Welcome to Hybrid Minds, unlocking the power of AI and IQ, the podcast that explores the groundbreaking concepts of combining artificial intelligence with human expertise to achieve unparalleled advances in various fields. My name is Wael Andonians. I am your host. I'm a data scientist, entrepreneur, and senior lecturer at the Frankfurt School of Finance and Management.
In 2020, I founded Cognize, an AI company that develops solutions for the financial services industry. Please join me in welcoming Jaan Tallinn to the show today. Jaan is the founder of Skype and Kazaa. He's an early investor in DeepMinds. which was later sold to Google. [00:01:00] His philanthropic activities include founding the Cambridge Centre of the Study of Existential Risk and other existential risk research organisations, such as the Future of Life Institute and the Machine Intelligence Research Institute.
Jaan is one of the initiators of the Open Letter. Paul's Giant AI Experiments, published by the Future of Life Institute in March this year. It was signed by over 30, 000 people, including Elon Musk, Steve Wozniak, and Yuval Noah Harari. Jaan, I appreciate you being here. I'm captivated by your statements. The value of AI is defined by the cost of its mistakes.
Before we delve into its relevance for businesses, could you discuss its potential implications for society?
Jaan: Thanks for having me. I mean, there are like, obviously, so many potential implications. so I think, I'm not the best person to talk about the entire spectrum. but [00:02:00] my focus really is on the catastrophic risks.
I think for the last hundred thousand years, we have lived in a situation where human minds. The minds of homo sapiens is the main future shaping force on this planet, and it's possible that in the next few years, that 100, 000 year period will end, therefore we possibly should embrace for like really big changes.
Vahe: I see. And do you see there? Also, opportunities for great things, or is it just a matter of risk?
Jaan: Definitely there are enough, opportunities in theory. The problem with practice is that the current AI paradigm, the large language models, they work by not building AIs, but by growing them in a way that... what I'm putting it is that like there is like an important difference between aliens and AI, sort of superhuman aliens or superhuman AI.
With superhuman aliens, we will have like no control over what kind of aliens we would get, should [00:03:00] they somehow find. However, with AI, we have, at least in theory, have the degree of freedom of, building or creating the AI that we would like, that would have kind of, some shared understanding and motivation about what is good to have in the future, what is, what is important to avoid from the future.
So we have this degree of freedom, but the big problem with current AI we are not exercising that freedom. we're not building AIs, we are growing them. The way that, frontier model development works is that you put something like 200 lines of code. I mean, there's like a much more in kind of, plumbing code, but like the conceptual code is just something, something like 200 lines.
You put them on tens of thousands of graphics cards and then leave them unattended for weeks, if not months at a time. This is not the humanity exercising control. This is humanity summoning alien minds.
Vahe: Yeah, that is very interesting, of course. And you know, I believe that a somewhat overlooked observation [00:04:00] is that we frequently encounter unforeseen outcomes with these large language models, whether they leave us, you know, underwhelmed or amazed.
Would you agree with this perspective?
Jaan: Yeah, sure. Basically, that's what I was saying. We are summoning alien minds and then like studying what can they do. I mean, that's what, The P in GPT means pre-training a generative pre-trained transformer. That's what GPT means and pre-training means means the thing that you do before training.
In other words, summing. So we have this someone and then train model of ai, and during this training mode, which. It comes after the author is someone. We try to make it behave and also like figure out what can it do and what it cannot do.
Vahe: Yeah, which brings me to the next point, because, you know, recent studies indicate that memory augmented large language models possess computational universality.
Do you think, are we overemphasizing this [00:05:00] Turing definition, or is it truly a groundbreaking
Jaan: revelation? Yeah. I mean, I don't know. I mean, obviously, like, if you have something that is Turing complete and has enough resources, as in like memory and clock speed, this can kind of, this is universal in the sense that, that you can, Use it to ultimately create universes.
it's like, we have never seen any process in this, in our universe that is not, cannot be perfectly approximated, with a Turing complete machine. In fact, like the full Turing completeness is not even necessary because like, you have infinite tape. You don't need infinite tape, infinite memory to approximate our universe, because our universe is finite.
This is actually a church called church. Turing thesis, that there is this thesis that there is like our universe, there's no magic in our universe. Everything can be simulated using a computer that is sufficiently powerful. So, so in that sense, I mean, for example, David Deutsch makes like a big point about this fact that humans seem to be Turing complete in a way that other [00:06:00] animals perhaps are not.
And, I think he kind of oversells this point in some of his podcasts that I have. Listen to where you are points out that like, well, humans are already touring complete. So we have nothing to fear from, from being joined by another species that is during complete, which I think is an interesting point, but I think it's just wrong.
Vahe: Wrong because?
Jaan: It’s still dangerous. Yeah. Yeah. I think it's like, I think it's a good point in the sense that like, there is no, probably no qualitative difference between humans and AI. Like there is probably nothing that AI can build that humans given like a billion years couldn't build. But there's still a really, really big difference whether, whether you need like one billion years to build this thing or a month.
Yeah, that's true.
Vahe: The progress that we achieved with this large language models demands substantial computational resources, right? It operates quite differently from our neural tissue. Do you believe that aligning more closely with the functioning of our brain [00:07:00] could decrease this computational power presently needed?
Jaan: I don't know, but my intuition is kind of the opposite, like, because like human brains are still, computationally. Kind of, I think it's, I think a similar ballpark that was needed for the training of GPT 4, I think, or like plus minus, another magnitude, but there's still, there's still definitely like much more powerful than any AI that is being currently running inference mode.
So, that kind of indicates that with only a hundred billion neurons, right?
Vahe: And if you think about it, like GPT, I think, four has 1. 7 trillion parameters.
Jaan: Yeah. Yeah. Yeah. I mean, I, this is something that I'm not, not an expert in, but again, like my, I don't, I don't think, I don't think human brains are like, efficient.
So it's, well. Efficiency can be defined in many ways. So it's, yeah, I don't think I should kind of say anything, very confident in that question. Yeah. I mean, there's an argument to bring, right?
Vahe: The [00:08:00] argument is that, if you look at the plane, right, the plane is using the same concept that the bird is doing with their wings, but it's much faster.So we could actually, have, have the same situation here. But on the other hand, if you look, we know. Really very little of how our brain works in detail. And I think that is one of the, of the major issues that we are facing right now is because it is conceptually as much as we know, very, very different.
And whenever we copied something from it, we actually benefited a lot. Here's an example, right? So if you look at this. Earlier neural networks, right, they process language in sequence, token by token, right? So inherently capturing an order of the words and, which is basically time, right? This transformer architecture introduced in this attention is all you need paper.
brought a significant investment, processing tokens at once in [00:09:00] parallel and doing this through a positional encoding. So basically, in a sense, we taught it to understand a position, not only time anymore, but also this position. And it
Jaan: resulted,
Vahe: this technology resulted in this explosion of language models, right?
So I think that there could be some. benefit into having a more closer mimicking of our brain, because that's what we are doing, right? In our brain, we have these grid cells that, you know, have a positional encoding and somewhat universal even positional encoding. The question is, If we have such a positional encoding, a universal positional encoding in neural networks, could that be the next thing?
Like could that lead to even bigger advances than we are, than we see right now? Or do you think it's really just computational power and, I don't know, training data at the end of
Jaan: the day? I mean, that's sort of like an eternal question, like that is kind [00:10:00] of highlighted by Rich Sutton's bitter lesson. where he kind of pointed out that, like, there are so many things that he can kind of think of.
And then you have, like, this, like, grad students just running circles around it, just throwing more compute at this thing and have it, have it discover even better, better approaches. Because I mean, we were ultimately, we weren't developed by evolution, right? So it's like an evolution had, like, no idea what it was doing.
It was, wasn't planning ahead, like, at all. So it's, ultimately there is no kind of like knocked on argument that. That you can't reach things with, with just throwing, throwing compute at it. But sure, it's, it's clear, clear that human brains are doing things that no AIs are currently doing. And it's possible if you kind of knew perfectly what.
What human brains, how human brains worked, in principle, what are the main principles, then plausibly we would already have superintelligence already be dead. Coming to
Vahe: superintelligence, when do you think, I mean, there's so many predictions out there when we are going to achieve AGI, do you have [00:11:00] any prediction
Jaan: for that?
So, like, one of my predictions that I currently still stand by is that, like, every 10x ing of compute that is being thrown at leading model development has, like, at least 1 percent chance of causing intelligence explosion. Or some other reason, for some other reason, causing things to spin catastrophically out of control.
So, it's like 1 percent is my lower bound, my upper bound is 50%, so it's like, I'm super, super uncertain. But I can't, like, honestly say that, like, if you throw, like, 10x, 50 more resources and then expect, like, less than 1%. You need, like, a very strong argument that, like, why, why you can't. You need a very strong argument because, like, saying that there's less than 1 percent chance after spending 10 times more resources than previously was unprecedented than, and saying that, like, nothing.
Nothing unprecedented will happen. And I don't think we have any kind of strong, such strong [00:12:00] arguments. And
Vahe: then when you say 10 times more resources, right? I mean, I don't know what your opinion or your stance is about Moore's law. But at least it seems like we are getting, either we are already out of Moore's law or we are very close to it, because at the end, we are not doubling anymore every 18 months or every, every, every two years.
So it's actually all the advances that we have done in the past is either through ASICs, right? So application specific, or. Or through packaging, right? So 3D packaging of things so that they're closer together, like Apple has done with this M1 chip. So do you think that there, that there is actually a hard limit that we can't, there should be a hard limit, right?
Because atoms have a size, right? And we can't go smaller than that.
Jaan: Sure. I mean, eventually we'll get into quantum computation and kind of reversible computation and things like that. But like pairing that, I think there are things like lambda ROI limit, like how much computation you can get done. with a piece of matter.
But my own estimate is that we still have like something [00:13:00] like six orders of magnitude, to go before we bump into, into fun sort of not fun, not even fundamental limits, but limits like that nobody has enough money to do that. Main places where currently people are getting those additional orders of magnitude.
are one, throwing more money at it, then second, kind of like changing architectures, making kind of more efficient use of calculation or like compute, packing things more closely. And the third one is just algorithm improvement. So I think those, those three components each can probably do like kind of two to three orders of magnitude by combined, I think we have like six orders of magnitude, which means 1 million times.
So we probably can. We can find ourselves training systems that are 1 million times more powerful than GPT 4 was. Yeah, makes sense. I
Vahe: mean, even, I think it was DeepMind, right, recently came out with a paper of faster, faster tensors, or faster matrix operations through [00:14:00] somewhat unexpected because you feel like the mathematics of it is clear and everything is clear, but then it's not that clear, right, at the end of the day.
You know, I think another, another pivotal advancement is in context learning, right? Because it allows a model to modify its behavior according to a given context, and it eliminates basically the need of an explicit retraining to change a behavior of it. And in essence, I think you could compare it even with social learning that humans are doing, because at the end of the day, you're not retraining anything, you're just giving it a context.
And, but at the same time, I think there are, while this is very fascinating, I think there are also lots of risks attached to it, right? Especially something, that you could be arguing that, at the end, because of the context, you don't actually know, it's not predefined what the output is going to be.
And hence it has an inherent risk
Jaan: that is not measurable. Yeah, I think that this is [00:15:00] something that I'm not super familiar with, but I think it kind of like, I just like file it under the general problem that is that like neural networks are, the current paradigm of neural networks just are black boxes that are grown, not built.
Therefore, like they're very hard to get actual guarantees about their performance. And that, And the performance failures, again, as, as you're gonna like, repeat that the, the value of AI or the dis utility of, of AI is kind of, mostly measured by the mistakes. By its mistakes or by, its more generally by its misbehavior.
And, so I think it's plausible that like, when humanity goes extinct because of. And then like aliens watch the tape, then they would still go like, oh yeah, I made a mistake there. Or like, definitely some people like Tom Dietrich or, or some kind of like prominent ML, ML researchers who don't believe kind of like AI that, that they would say, oh yeah, that was like a dumb thing that AI did.
Like why did it build an AI that did a dumb thing? Yeah, but you [00:16:00] are,
Vahe: but because you are putting a lot of your effort, your time, your money into, you know, awareness on this issue. So you do believe that, we can actually stop that, right? We can actually regulate it and we can actually make sound decisions that will result in not having such a, such a disastrous
Jaan: outcome.
I mean, at the, at the very least we can try, like, there is like nothing that stops us from trying, especially like there is like more and more. Interest in regulating things, both in, in among politicians as well as in a voting public. just recent YouGov study said that was like overwhelming and bipartisan majority in US.
I think it was like Vox article recently that, that headline was that the public attitude towards AI can be summarized with two words, slow down. Yeah. Yeah. I mean,
Vahe: it's also economically a huge problem, right? I mean, I think you can compare it with an industrial revolution. We had a couple of those already.
And every time, even though the end outcome [00:17:00] was a good one in the past, the transition period was very hard. And I think this could be equally hard. I mean, if you just look at what's happening in the job market, I think it's already...
Jaan: Yeah. Yeah. Although, like, I point out that in this... Whenever you're kind of like thinking about, unprecedented situations, you have the risk of picking the wrong reference class.
So like there's a kind of famous point, that some people have been making is that like, sure, industrial revolution didn't, or like the introduction of automobiles rather didn't kind threaten human jobs, but like the horses are gone so it's or like at least from their economic, they eliminated from their economic function.
In some ways, if you treat AI as new technology, you might make the mistake of treating humans as humans rather than horses, as we should in this context. That's
Vahe: true. That's very true. That's a very interesting point. You're very right. It leads, though, very much to a very much [00:18:00] to a bad future picture, right?
So it's
Jaan: like, it's kind of unsurprisingly, because like, if you think about all possible reachable futures in terms of like where the atoms can be in the future. Like, because there's so large possibility of where those atoms might end up with, almost none of those futures contain human shaped figures.
Because just, just like the number of, futures that contain humans is just so small compared to things that, that can be reached. So in some ways we, when we want to build a better future, we are aiming for a very narrow target. We are, we are aiming for increasingly narrower target. As the kind of guardrails come off when it comes to technology, I mean, when we are developing things like stone access, like there's only so much damage you can, you can do.
By developing a bad stone axe, there's way more, way more damage that you can do when you're developing a bad cannon and then like bad nuclear weapon. In general, like that effective [00:19:00] radius, effective leverage that technology gives humanity and even individual humans just keeps increasing. Whereas our planet doesn't keep increasing in radius.
Vahe: True. Before you touched on an interesting concept, you said, like, neural networks are basically black boxes, right? And what I'm asking myself in this issue, in this, in this topic is this, that, yes, they are, but so are humans actually, right? Because if you think that, if you think about, you know, Daniel Kahneman's Strangers book or other research that he has done, we don't even know why we come up with positions and it's very hard.
It's very complex. So, we ourselves don't even know how we work,
Jaan: right? That's no way. Yeah. And we are very bad for other species. And that's true.
Vahe: That's true. Right. So, so why would we expect that we ever understand artificial intelligence if it is a rudimentary copy of, of the human
Jaan: brain? I mean, it's not a copy.
It's like, it's very different, in, in multiple ways, like one very important reason Oh, the important way how it is, different [00:20:00] from human brain is that you can kind of like opening, open it up and look what's inside and put in like various things to measure it and, and analyze and dissect and kind of like reduce and, and distill and do like all sorts of things that like neurosurgeons only can, or neuroscientists can only imagine, only can dream of.
So it's like much more accessible in, in, in, in some sense also is trained very differently. Like, human ha humans have this genetic bottleneck, like everything. between human generation has to be squeezed into something like 15 megabytes. that is like the size of the compressed DNA. Whereas like, AI has this like, I don't know, up to trillion parameters, and there's no generational bottleneck, like everything that is like modified here is directly modified, not just this bottleneck 15 megabytes.
So you modify all this, potentially modify all this trillion, like a terabyte basically, whenever you are, you are doing a training step without needing to compress it. So like they're kind of melee. Significant differences, but like, yeah, unfortunately the big similarity [00:21:00] is that we are both black boxes.
Vahe: So what is your, what would be your suggestion? Is it regulation? Is regulation the answer? Is it the moratorium? What is the answer to?
Jaan: So it has to be a combination of, Sort of the famous academic slogan, more research needed, because like we really, even though like we have had kind of warnings from like, like from the people like Alan Turing from 70 years ago, more than 70 years ago, he said that like, once AI becomes as powerful as humans, we should expect to lose control to it.
We didn't really spend much of that 70 years researching ways how to. How to remain in control or how to make sure that the future goes well with AI that is potentially more powerful than humans. So we should catch up. We should do more, more research. But in order to do that research, we need time. And in order to get more time, we need pause or constrain the top AI experiments, the top models summoning process, [00:22:00] because this is what currently is dictating the pace of everything really.
So, and in order to. Stop or constrain or pause these experiments if we need some kind of regulation with, and there are a bunch of, like, good ideas, like one obvious idea is that, like, AI experiments, AI, big AI experiments should only happen in, in designated data centers that are licensed and certified.
And then you can start, once, once you basically have, once you constrain the training experiments, then you can start kind of adding more and more constraints in terms of security and, and safety. Thank you. Don't
Vahe: we then have a little bit of a prisoner's dilemma, because if you constrain yourself, it doesn't mean that others do, right?
Especially if it's regulation, then it's maybe countrywide, but doesn't mean that other countries
Jaan: are doing it. At this point, like, a few things that I can point out, like in theory, yes. So that it cannot be like, cannot be like long term solution in the short term, though, like China is like ahead when it comes to regulating AI, like [00:23:00] that's the only country that has AI regulation at this point.
EU is trying to get there, like hopefully later this year, we'll see. US hasn't even started. So it's, like, in some ways kind of like the but China, argument is correct in the long term, not so in the short term. And the other thing is like the chips, like there is only like current. Dominant pipeline that goes from, from Holland manufacturing the machines to Taiwan manufacturing the chips to US assembling the cards that NVIDIA sells to cloud providers.
So and this is like very legible, very easy to track at least in principle pipeline that that doesn't really have this prisoner's dilemma unless we don't do anything about it. Yeah, true.
Vahe: Yeah. I mean. Maybe as a last point, you mentioned NVIDIA assembling it. Do you see there any, any risk in like one company dominating the compute?
To that degree, because I mean, I think there's no compute basically possible without NVIDIA right
Jaan: now. [00:24:00] Sure. I mean, there is like this general question about centralization risks versus like existential risks. The problem with centralization, as like many people are quick to point out, is that you can have like this power imbalances that traditionally have like cost.
Bad things like, like kind of tyrannies and whatnot, but then there's like the other problem, which is like other asymmetry, which is like destruction is easier than construction. So like, if we find easy way to construct nuclear weapons in your garage, like should we kind of like make sure that everyone has the capability of doing so, or perhaps not.
Good point.
Vahe: Jan, it was a great pleasure. Thank you so much. Thank you for your time. Thank you for your insights and thank you
Jaan: for your work. Thank you very much. Thanks for listening to this edition of Hybrid Minds. This podcast is brought to you by Cognize, the first of its kind intelligent document processing company, which automates unstructured data with hybrid intelligence and a data [00:25:00] centric AI platform.
To learn more, visit Cognize. com. And be sure to catch the next episode of Hybrid Minds wherever you get your podcasts.