Welcome And Guest Background

SPEAKER_00 0:14

Welcome to the Where Shall We Meet podcast. Our guest this week is James Manika, who is Senior Vice President of Research, Labs, Technology and Society at Google and Alphabet. He joined Google in 2022 in a newly created role, reporting to Sundar Pachai, and has spent the years since arguing that AI's largest contribution may turn out to be science itself.

SPEAKER_01 0:38

His latest essay in the new volume of Daedalus, the Journal of the American Academy of Arts and Sciences, is devoted to AI and scientific discovery, which features Demis Acebis, Jan Lakun, Yoshi Abenjo, Eric Topo, and others. He also co-chaired the United Nations Secretary General's High Level Advisory Board on AI, served as vice chair of the US National AI Advisory Committee, and co-wrote No Ordinary Disruption.

SPEAKER_00 1:04

James was born and raised in Zimbabwe and read Electrical Engineering at the University of Zimbabwe before going to Oxford as a Rhodes Scholar, where he took a doctorate in AI and robotics and became a Fellow of Balliol College. He was a visiting scientist at NASA's Jet Propulsion Lab, then spent 25 years at McKinsey, 13 of them as chairman of the McKinsey Global Institute.

SPEAKER_01 1:27

We talk about Alpha Fold and the 50-year grand challenge.

SPEAKER_00 1:31

Did it understand how proteins fold or just learn the grammar?

SPEAKER_01 1:35

Is hallucination a bug or a feature?

SPEAKER_00 1:39

Move 37 and the bitter lesson. Is this the first time we have had a cognitive competitor?

SPEAKER_01 1:49

The impossible problem of alignment.

SPEAKER_00 1:52

AI as an invention for making inventions.

SPEAKER_01 1:57

And who actually benefits from a breakthrough? Let's discover. Hi, this is Amida Shari.

SPEAKER_00 2:06

And Natasha McElhone. And with us today we have James Manico.

SPEAKER_01 2:11

Hey James. Thanks for taking the time.

SPEAKER_02 2:13

Oh well, thank you for having me. I've been looking forward to this.

SPEAKER_01 2:17

Likewise, I wanted to start us off by

AlphaFold And The Protein Challenge

SPEAKER_01 2:20

referring to your essay that you've written, which is covering a large swath of how AI is helping with scientific discovery right now. We'll have a link in the show notes for our listeners. And the way we wanted to move into this topic was maybe starting with AlphaFold. Because I think it's something that people have heard and I think it's a good entry into this. So the thing about AlphaFold is that it didn't really simulate physics, it didn't learn the world from first principles. It kind of learned the grammar, so to say, and then skipped the whole theory and came up with all these proteins. I think it would be really interesting to talk about what that means for understanding. And maybe you can talk us through what AlphaFold is and how it is different from what we've done before as in science.

SPEAKER_02 3:05

Yeah, the story of Alpha Fold in some ways it's pretty interesting because it's emblematic of the kinds of uh contributions in science that we can see from advanced AI systems. I think it's it's worth going back to the to almost first principles on this. First of all, uh proteins are the fundamental, one of the most fundamental building blocks of life, uh of biology, of understanding, and protein structures are the result of uh amino acid sequences and how those amino acid sequences kind of curl up to create these structures of proteins. And it was known for a long time that in fact it would be nice if there was a way to computationally predict the structure of these proteins, because if you're not able to do it computationally, all what you're left with is doing lab experiments, work lab experiments, x-ray crystallography, one protein at a time to figure out the structure of each protein as you go. So back in 1972, a Nobel laureate, uh, Kristen Amphinson had made the case that there's got to be some way to do this computationally. This is gonna be one of the most important grand challenges. So the question of how to do this computationally kind of stood there. Uh, meanwhile, humans were painstakingly, painstakingly figuring out the structure of each protein at a time. And before Alpha Fold, uh, thanks to the extraordinary work of the uh Protein Biobank, we'd actually come to a protein database, we'd actually come to understand the structure of something over about 150,000, 170,000 proteins doing this painstaking way. But we know that there was something like over 200 million proteins known to science.

SPEAKER_00 4:48

And just give us a timescale of how much each one of those took to uh sort of ratify, I guess, or b before Alpha Fold.

SPEAKER_02 4:57

Well, it's generally sort of understood that the rule of thumbs that it takes something like a researcher or PhD team at a in a lab, something like three to four years, to work through the structure of each protein uh in the kind of traditional extra crystallography kind of way. So this is very painstaking work. So thanks to the work of many, many scientists over many decades, we did have the structure of something like 150,000, 170,000 proteins. So the so the challenge still stood there, which is considered a 50-year grand challenge to say, how could you do this computationally? So there's always this competition called Caspus, which every year, people, every two years, people would get together to try to figure out are we making any progress? So to cut a long story short, what Alpha Fold did was basically, the first AlphaFold basically learned to predict the structures of proteins from the uh protein structures that were already in the protein data bank. Alpha Fold 2 did something better. It actually tried to learn uh uh on its own, uh, including making up its own predicted

Predictions Versus Understanding

SPEAKER_02 6:05

structures of proteins to figure out what the protein structures were. So it was able to predict the protein structures of over 200 million proteins quite accurately to the angstrom level, uh, which is the kind of accuracy you need to have to be able to make these be useful predictions. So at this stage, uh, you know, uh uh something like over 100, over 3.3 million biologists around the world, you know, over 190 countries, are actually openly using these predictions. So now when you're trying to study a protein or trying to understand a particular disease or develop a therapy, you don't have to wait until you figure out the structure of each protein. You can just look it up uh from the AlphaFold database. And then I mean what a gift.

SPEAKER_01 6:52

What a gift from those scientists from years before, and what a gift from AlphaFold to the scientific community that follows, right? That's quite exciting how this is a circle that closes. The question still remains: does it just learn the grammar or did it really understand how proteins fold?

SPEAKER_02 7:08

Well, that's the question, right? So that's also one of the questions at the heart of science in many ways, which is you know, do you just care about making accurate predictions that accurately reflect reality? If that's the question, then you'd say, yes, AlphaFold can accurately predict the structures in a way that we can validate in real life. Question is a separate question, does that mean AlphaFold understands the grammar of proteins? Uh that's why in the essay I put it in quotation marks, because it depends what you mean by understanding. If you said, can we now write equations or uh chemical processes or uh equations that describe how protein structures evolve, then you'd say probably not really. So the question of what does it mean to understand protein structures is still an open question. Uh but you know, are these predictions accurate? Yes. Uh they can we use them reliably to do science? Yes. Uh so in fact, we may actually get this a wider divergence here that will only get wider as we proceed, where we develop models that are predictably predictably useful uh and predict reality and help us predict and make predictions and so and model the world. Then there's a separate question. How do we actually understand what that means and what they're doing? Because scientific discovery, historically at least, has always relied on the fact that if you make a prediction of the world, hopefully, along with it, we have a deeper understanding, either of the underlying mechanisms or equations or theories that are the basis of that prediction model. But in this case, we don't quite know that yet. In the case of Alpha Fault. Is it still useful? Absolutely.

SPEAKER_00 8:57

I get that it may or may not understand, and that in the short term that doesn't matter so much because the progress is being made. But if it's all predictions, surely I mean predictions aren't facts because facts are only things that have already happened or things that are happening now. So given that a prediction is something that's in the future, surely you don't really know whether it's going to work until quite a lot of time has elapsed.

SPEAKER_02 9:24

Uh well, if you make a prediction, there's a few ways to validate it. Either you wait for it to happen or you do a lab experiment. Uh to see is my conjecture, my structure. In fact, we've actually seen this, for example, in the case of uh material science, for example, where we now have to be a very important thing to do.

SPEAKER_00 9:45

Yeah, sure. I I know that these things have led to incredible breakthroughs, but it's just this the whole system being reliant upon probability or prediction. Well, it can't be. And I know we don't have anything else.

SPEAKER_01 9:57

We can't be, no. I mean, when if we don't have the data, we can't really predict. So there are areas where there's very scarce data and we can't make predictions, right? And that's still the domain of humans. But in these situations, and I think what James was saying

Ontological Inversion In Modern Science

SPEAKER_01 10:10

is that you're referring to this as the ontological inversion. You have essentially these simulations that are giving us predictions, and then what we do in the lab is validate this. Well, it was actually the other way around before, right? Maybe you want to further.

SPEAKER_02 10:23

Because historically, the way you know science is historically proceeded is that either you kind of has relied on these two pillars. On the one hand, you start with uh, you know, theories, logic, uh, kind of some view of how you think nature works, uh, either theoretically or because you have intuition. And then you do the other thing, which is you then empirically test that to see does that match observations or does that match reality? And so this interaction between theory and experiment and observation was the way in which you validated things. So to the question you were asking earlier, Natasha, is that so if you have a theory where you make a prediction, you either wait for it, see if you can observe it, or you test it in in the real world. Does it match reality or not? If it does, then presumably you'd conclude that that theory must be right, because it seems to match the observation. It's a bit like when when Einstein first came up with special relativity. For a long time it wasn't it wasn't actually validated empirically until there was a Michael Morley experiment in 1919, where the idea was to actually see if, in fact, uh gravit gravity would actually bend light. And until we'd actually observed that, special relativity was just that, a theory that hadn't been empirically validated. So if these models are making predictions, uh, you either wait to observe them or you validate them empirically somehow in the lab, observing something reality. So that's the question, but it still leaves open the question around okay, just because the the predictions are correct or they match reality, does that mean we then understand what the scientific underpinnings of that are? And that's an open question. So alpha fold is validated in the sense that the prediction for protein structures do match reality. That's why they're useful. But does that mean we understand the mechanisms by which proteins fold, uh, the so-called grammar of proteins? It's clear we do.

SPEAKER_01 12:32

Right. And this goes to what uh I think makes Techmark says as well. If you and I think you referred to it maybe in in the essay as well, is like uh if you can predict the motion of planets to 99% accuracy, that may be more engineering rather than real understanding.

SPEAKER_02 12:48

Exactly, exactly. And so you often have, in fact, this is this is an interesting creates an interesting dichotomy because people just focusing on focusing on building useful things may be satisfied with predictions that match reality. They work, they allow us to model the world, they allow us to predict the world, they allow us to design things, they're tools for engineering. Uh, but there's still a separate question about does that mean our understanding has been enhanced just because a model is accurate? So these become two separate things. And I think this may be a question in the future, which is because up until now, we've been able to keep these two things roughly hand in hand. When we have models or mechanisms to make predictions, we've also

Hallucinations As Creative Leaps

SPEAKER_02 13:35

been able to create the theories and understand how those things actually work. So the two have kind of gone hand in hand.

SPEAKER_00 13:42

But what I was interested in in your paper was the suggestion that nothing's inevitable. So yes, there are these predictions and likelihoods and then they're tested and they're verified. But when you spoke about the hallucinations, which has had hugely negative press, in this context, those could actually be very productive because they were outside of anything that we were able to predict. So I suppose for me there was like a third possibility that you could enter into this hinterland that isn't human, but that then, as we were saying before, you need to have a sort of interlocator to be able to translate whatever it was that was being well explored, if not discovered, and then sort of fed back to us. And then that leads to greater questions around whether we then become the minions and we don't really understand anything that means, which is I know the big fear. And the other thing about predictions is just are we then freighting all of our resources into something that we've decided already, you know, is important? You talk about representation a little bit as well in the paper. I can't remember who it was who had a particular paper or link to that, you know, like diseases in Africa, for example, which perhaps didn't receive nearly as much attention as Parkinson's or Alzheimer's.

SPEAKER_02 14:59

Yeah, no, uh that was a paper by Kelly Chibale who was talking about, he wrote a fascinating paper just on drug discovery in Africa and how that enhance uh uh possibilities in Africa. But I think I want to go back to the I think what you what where you started, uh Natasha, which is hallucinations are a fascinating thing in the case, in the context of science, in the following way, because typically we people worry about hallucinations in AI systems when you're looking for accuracy. So when you're looking for accurate facts, accurate data, and you don't want hallucinations. But at the same time, uh hallucinations can be a feature as opposed to a bug in the following sense. When you're trying to explore uh unrealized territory, uh and you're you're trying to be creative, actually, hallucinations uh uh can be a feature in that sense because they could point you in directions that are interesting that you hadn't thought about, that might spark ideas. So the the example, uh I mentioned a few examples in the paper, David Baker, for example, who also got a Nobel Prize at the same time around proteins, uh, he actually attributes some of the leaps that he made from a few hallucinations that then pointed him in directions he would not have looked at. And then when he did look there, it was able to inspire his own creativity to come up with interesting designs. You often see this, for example, uh in even in creative work, when when creative artists are using AI systems. It's a bit like, you know, when you're trying to have a system that makes a prediction, peanut butter and in America you say jelly. Now, if you're trying to write a creative sentence, you may not want peanut butter and jelly. You may want a more imaginative version, peanut butter and marmalade of something that's creative. So a a system that's trying to make an accurate prediction would have probably come up with peanut butter and jelly. But as a creative person, you may not want that. That's too cliche. You want to go outside and see something else imaginative. I'm reminded of something that um actually Alan Turing said, which I thought was quite fascinating. He actually made a comment that you don't want intelligent systems to be infallible. Because if they are, they're not very interesting. You actually want intelligent systems. Intelligent systems are preferred if they can make these intuitive, inductive leaps, make mistakes, uh, and you know, and come up with these creative things. As long as they can learn from them and explore those possibilities. That's actually what you want. If they're too infallible, then they're they're not going to do anything interesting. People have said the same thing in science as well, which is, in fact, there's this other tension which I explore in the volume, it is characterized by uh Rich Sutton, who's a Turing Award winner, described it as part of his what he called the bitter lesson, which is that every time we've tried to infuse

The Bitter Lesson And Move 37

SPEAKER_02 18:05

AI systems with what we know, what we think are the theories, what we think is the science. Sure, we get some short-term incremental improvements, but we miss out on these big leaps because we're constraining these systems with what we know, with what we understand. Why not let them explore broadly? They might even get to territory we don't understand.

SPEAKER_01 18:28

That's the AlphaGo Zero that's right.

SPEAKER_02 18:30

That's the AlphaGo Zero example with move 37, because all the all the humans who'd ever played uh AlphaGo for millennia thought you don't play that way. You should play this way. But AlphaGo went somewhere else that human players had never gone before and came up with Move 37, which generations of AlphaGo players would never have come up with. In fact, they all thought this was a bad idea to play this way.

SPEAKER_01 18:57

I think the the other uh point, maybe to drill in here a little bit, which illustrates this friction to an extent, is of course we can make really great predictions when there is ample data available, right? But the boundaries that we're facing very often are those where we don't have enough data. And in those boundaries, so far, the Einsteins and the Newtons and you know other people who just ran a counterfactual, had an intuition, had a hunch, which is exactly what you're saying, a high entropy state in their brain, which is maybe a hallucination and high temperature hallucination, um, was the thing that was generative that pushed things forward a little bit.

SPEAKER_02 19:34

Yeah, but but I think even that has a there's an interesting dichotomy. I think there's a there's a whole history, and you know, including the current state, where you can make predictions based on how good the data you have is, how broad it is. So you're making kind of within distribution predictions within the distributions of the data sets you have. But often you want to go outside of that. And you're starting to see ways and experiments and mechanisms to go outside of the data you actually have. So, for example, that's why you see the growth of things like synthetic data creation, for example. In fact, this was a little bit of the case even in in Alpha Fold II, uh, as opposed to Alpha Fold One, where Alpha Fold II actually made up a whole bunch of other uh training data sets that were not real data, but synthetic data as a way to expand its own training possibilities. So even the question of data, while it is the case that most of the progress that anybody will point to so far, most of it, is based on isn't in is in domains and environments where we have a lot of data, that is the case. But you're starting to see you know breakthrough innovations that allow you to go outside of that, either create new data sets or work on other kind of latent representations of things as opposed to actual empirical data.

SPEAKER_00 20:58

Aaron Ross Powell But was that also Richard Sutton, who was talking about um that a truly scientific AI would have to be able to go out into the real world rather than be limited by LLMs and reaching a dead end?

SPEAKER_02 21:11

Yeah, so so there are uh a number of people who think LLMs are a dead end in the sense that until you go and experience the real world, and uh that's the big push of about this idea of building world models, people like Faye Fei Lee and many others who are thinking about this think that the next set of breakthroughs will come when we have systems that actually go out into the world uh and experience the world and collect and gather and observe the world. That's the basis for breakthroughs. But some disagree with that. Uh, some would say even the systems we have seem to have been able to learn, for example, physics just by watching videos. If you're

Data Limits Synthetic Data World Models

SPEAKER_02 21:55

watching our videos of objects falling, uh that may be sufficient to actually. Start to understand that when objects fall, this is how it seems to work without actually having seen in the world real world objects falling.

SPEAKER_01 22:10

This is also what Demis uh I think Demis and Dario are basically on the we can scale this up all the way to um you know. Well, not not the same way. Not the same way. Yes. Uh yes, I think Demis says there are breakthroughs, but I th they need we need some breakthroughs, right?

SPEAKER_02 22:25

Yeah, so Demis's view, I think, you know, he who he represents it well in the conversation I have in the volume with them, is that I think we're gonna need his view, which is similar to mine's, I think we're gonna need both. Scaling will get us some gains, and we're not done with scaling yet. But in addition to that, we may need a few more breakthroughs because we know some of the current limitations of the current scaling mechanisms that we're using today. Uh, but there are others, and Dario maybe in this camp, who think actually we can scale all the time.

SPEAKER_00 22:55

And is that enough? I mean, is that not the sort of meta-question of like a lot of human beings, they they don't want embodied AIs. They are quite happy for for human beings to have the exclusivity on that and for AIs to remain in a machine because they're already so competent.

SPEAKER_02 23:15

Well, that that that's part of the tension. Uh I could argue, I'm a I live in San Francisco.

SPEAKER_00 23:20

Um, I know you're excited by it. That's why I want I okay, I guess cars, and there's lots of yeah, automated.

SPEAKER_02 23:25

Well, well, car cars. I mean, we we you know, we've I've been living in San Francisco for the you know, for the last few years we've been living with the reality of embodied AI systems, which are driverless cars. That's what they are. They are AI systems on wheels, navigating and moving through the world and doing fairly intelligent things. Uh, so that's our first real experience so far, at least, of embodied AI systems. Now, in the future, we may have other systems, humanoid robots, and all kinds of things. But today, so drones already, right?

SPEAKER_01 23:58

As well.

SPEAKER_02 23:59

You could argue drones are too. So I think the possibility, in fact, I I I I actually suspect that part of the human anxiety, uh, I think maybe historically less about embodiment and has been historically more about the cognitive part. Because I think we've been quite comfortable with artificial or mechanical systems doing physical things in the world. I mean, we've had electric drills, we've had power tools, we've had bulldozers. You could argue those are physical embodied extensions of things we can't do.

SPEAKER_00 24:37

But I think they're not multi-purpose, are they?

SPEAKER_02 24:40

Well, they are, but but I think the difference, the the the discomfort that I sense most people feel is the fact that this is the first time we have a cognitive competitor.

SPEAKER_00 24:50

Yeah. And it's a genetic. I mean, if it can make the decisions about what it's going to do and it's embodied, then it has everything 100x compared to. I mean, it's a comparative thing, no?

SPEAKER_01 25:03

Until you turn off the battery?

SPEAKER_02 25:04

Until you turn off the battery. Well, this is actually one of the things one of the people I quote in the in the volume is uh is Joshua Bangio. Joshua's one of the uh uh Turing award winners, one of the so-called uh godfathers, three godfathers of machine learning or uh uh of deep learning. And he's always argued that uh, you know, until we solve questions around alignment, we may want to uh restrict our AI systems to mostly kind of uh epistemic tasks, oracle AIs, where these are systems we consult, so they tell us, but we then go act in the world. Uh but uh, you know, because you know, there's the concern is that if you're gonna go actually act in the world, you boy, you better make sure we've solved the alignment problem. Uh, because otherwise you don't want uh systems taking actions when you're not fully understanding what those actions are and on what basis they're being made and what those decisions are being made. So alignment is actually one of the really big

Embodied AI And Cognitive Competition

SPEAKER_02 26:05

and important questions with these systems.

SPEAKER_00 26:08

Going back to, I suppose, the hallucination possibility is that what happens if their decisions are much better than human decisions? I mean, we're not great at being aligned between ourselves because we're constantly at war and committing war crimes and even laws and parrot success.

SPEAKER_01 26:24

Self-driving cars are not gonna get drunk on the road, right?

SPEAKER_00 26:28

W why isn't this um pioneered a little more? I I constantly hear that, you know, there's lots of nail biting around this and people are very apologetic. Yes, yes, yes, we've got to make them aligned with humans. I mean, I'm being naive, I I'm sort of playing devil's advocate, but we are so in conflict, we're so dreadful to one another, it it's almost hard to imagine that a machine that has been trained, you know, relatively carefully would behave as despicably.

SPEAKER_02 26:53

Well, well, I think I think the the alignment question in this regard, Natasha, kind of has two aspects to it. Uh there's alignment in the sense of, you know, do they re do these systems, will these systems reflect human preferences or human desires? And that's a fraud question in the sense that, okay, which human desires, which preferences? Uh do you mean yours, Natasha, or mine uh, or or meads? Or do you mean those of this community or this country or this group of people? So there's always been that, you know, millennia-old question as to which is easy to state, which is to say, align these systems with human preferences, and my question will be which ones? Uh so there's that, there's that problem. I think there's a separate problem, which isn't really about alignment with preferences one way or the other, but is the question that often causes some existential anxiety, which is what if these systems become misaligned in the sense of coming up with their own goals, their own preferences quite separate from us. I mean, this is the the the kind of the, I think, my view, low probability possibility that leads to kind of the typical sci-fi scenarios uh of humans losing control. So I think you've got these two questions. I think the first one is is a human question. It's a very hard one. I don't think we've solved it for millennia. Uh it's really a question about for us as humans, which ones, which preferences, which values, which desires should the system, even if I'm talking about just me, do I want the system?

SPEAKER_01 28:33

You're not self-aligned to be. Yeah, I'm not self-aligned.

SPEAKER_02 28:36

Do I want it to to do I want my AI to do what I ask it to do or what it observes me doing? Which is the better.

SPEAKER_00 28:44

The truth versus your yes, exactly.

SPEAKER_02 28:46

Well, because often versus action the things I say are not the things I do. No, no, no. I want to be fit, I want to eat healthily. I don't always do that. So if my AI observes me eating unhealthily, maybe he thinks that's what James would like to eat. So we'll do it.

SPEAKER_01 28:59

Somewhat ice cream for James.

SPEAKER_00 29:01

But that's before we even get to religious differences, sensibilities, absolutely. I mean, there's a whole set of people.

SPEAKER_02 29:10

And in fact, the alignment question has another dimension too, which is because often is the there are these kind of normative questions which have to do with yours or mine or this community or that community or this country, that religion, this culture, etc. Then you've got questions of plasticity, which is what I may, what if we build systems that are aligned with what we think today, are we going to think that a decade from now? Imagine if we'd had AI systems built in 1950, when at the Dartmouth conference that coined the term AI and they're built in whatever the values and views were in 1956. Uh, and now we're in 2026. Would we still like those preferences and those values? Imagine if they've been baked in for all time. So there's also these questions of plasticity and change over time when it comes to the process.

SPEAKER_00 30:02

But that's why you have the self-improvement being recursive, right? But that's then what creates the risks of them running ahead and not being aligned with us.

SPEAKER_02 30:11

I think self-improvement has its own challenges because in the case of self-improvement, this is the, I mean, this is not a new idea. IJ Good kind

Alignment Preferences And Control Risks

SPEAKER_02 30:20

of framed this back in 1964. Is the question of, so let's just imagine we're able to build systems that are so good that they can also build themselves. So they can build their own successes. And if they're really good, I mean, this is often what people then worry about is you might have a runaway situation. So self-improvement has both extraordinary benefits and extraordinary challenges. The benefits are imagine if Alpha Fold was self-improving. Now it's understood proteins. It goes on to understand cells, it then builds its own successes that can understand all these other questions that we don't know how to answer today. From a scientific standpoint, you'd say, that's incredible. Let's do that. But from a risk standpoint, is if you then build systems that you don't fully understand and you haven't fully aligned, and then they start to build their own successes, then you start to worry about safety and you start to worry about control. So recursive self-improvement is this extraordinary, could be incredibly amazing for scientific discovery and progress, but could also raise these extraordinary uh risks and challenges if we have unaligned systems.

SPEAKER_01 31:34

I think you point to this in a different guise in your essay, which I thought was quite unexpected. And that was the point where you said when we become the students of the machine's outputs, when it is, you know, self-recursive improvement takes it to a place where we can't even catch up anymore, where we need maybe translators to give us lessons about what the AIs are figuring out, or whether, you know, we're just taking the observations of those things and we don't even understand the underlying things, we just go with it, whatever it says, because it's working. We don't understand the universe anymore, but our lives are getting better at something. And and the question around meaning there as well. Um, so that that I think is bundled up with this point.

SPEAKER_02 32:16

Yeah, it it is. I mean, I think uh to get to that situation, I mean, I don't think you necessarily need to have uh recursive self-improvement. I'll give you a good example. And Mario Kren, in one of his essays in the volume, points to this where, you know, we already have examples of AI systems doing actually breakthrough things. Uh and and we can understand how they did them. And we don't quite understand what's going on. We know they work because they match observations and predictions. Uh so the question is, you know, is that an alien form of intelligence? So could we actually interrogate the AI systems to explain to us what they did or how they came up with that intuition and insight? So we may already have been in that situation where we now have these breakthroughs. We don't understand what's going on, but they they're clearly breakthroughs. You could argue Alpha Fold 2 is an example of that. Uh, it clearly must have some view of the grammar of proteins. It just hasn't told us yet.

SPEAKER_01 33:24

And I guess we're doing this with weather, uh, you know, with weather forecasting. I mean, us humans, we prefer we forecast weather pretty reliably to an extent and we don't know how it works, right? So we do this ourselves. Yeah. And now AI is doing it even better, and we still don't know how it works. Yeah, no, it's it's a good point.

SPEAKER_02 33:39

It's a good point, I mean, because this is not a new problem, right? And we've had this for with weather, with uh stochastic systems, or there's any number of these kinds of systems where the predictions work. We can't explain them. So we've had these kind of black box uh issues, epistemic issues for quite a while. The chat the potential here is they could get bigger and even and more pervasive. And then you get to the the idea that are we gonna then need teacher AIs, which then take all these breakthroughs and then explain them back to us.

SPEAKER_00 34:16

Is there an endeavor to get an answer about how it came up with this? Because you guys can't figure out how it found that solution and it doesn't seem to know through a series of uh interrogation. Let's just say you got a an answer that was satisfactory, then you'd feel able to move on to the next level of complexity or freedom, allowing it to explore further in a slightly unboundaried or not so limited way.

SPEAKER_02 34:41

Yeah, I mean, the it's actually one of the big kind of areas of research, which is to try to understand what's actually going on inside these systems. Uh it's it comes with so many different terms, mechanistic interpretability, uh, and a whole range of terms. And uh so today what what mostly we do is to do look at these so-called reasoning traces to ask the LLMs or these foundational models to say, explain your sequence and steps as you go, or tell us how you did this. Now that's

Recursive Self Improvement And Teacher AIs

SPEAKER_02 35:15

that's still an open research question because take the human version of that. If I ask you, Ameed, Ameed, you just made a big decision. Explain to me how you made that decision. We know with human subjects what they tell us may not necessarily be what actually happened. Often there's a bit of back rationalization and so other kinds of things. So there's no guarantee that what Amid might tell me is exactly what actually happened.

SPEAKER_00 35:43

Sure, but they're not human. So I would have thought that the ability to trace back would be far more predictable.

SPEAKER_02 35:50

But but I think that's exactly the challenge, which is because they're stochastic probabilistic systems, uh, there isn't a linear algorithm that's running that I can go and rerun to say, you know, let me rerun this to understand what actually happened in the system. So because these are probabilistic systems, and as we scale them to be much, much larger, the problem only gets worse.

SPEAKER_01 36:16

And there's one thing to say if it does something that is language-based, it's easy for it to explain. But if you're doing something in the world of combinatorical sequences, cell starting or like amino acids or whatever. Exactly. You're doing things in 50-dimensional spaces to compare and find patterns. It's very hard to then explain that to us in some sort of language that we wouldn't necessarily understand, maybe.

SPEAKER_02 36:41

Yeah, so in fact, you know, one of the most more fascinating essays or contributions in the volume is the composition that I had with Jan Lakun. Uh so Jan, who's one of the um three so-called godfathers of uh deep learning, uh, he's spending a lot of time trying to understand these kind of latent abstractions and representations. If you're dealing with these large spaces, uh uh think of these as, for example, cell structures or these kind of at the things at the mesoscopic scale, where we don't even quite have intuitions about those things or even ways to talk about them. You find the same thing, by the way, in quantum data, in in multidimensional quantum states, where our intuitions fail very quickly. So it's not clear that having an AI system explain the, you know, uh would be any good to us anyway. We may need a translator. Uh back to almost teacher AIs. How do you translate these highly abstract, uh, you know, multidimensional space back to dimensions we understand?

SPEAKER_00 37:47

And how much do you personally mind understanding or not understanding? I know how curious you are, and that your curiosity would be chasing and chasing and chasing and trying to find out the genesis of how that happened. Would you rather have the discovery and the exploration without the so-called security?

SPEAKER_02 38:05

I actually like both. Uh so the practical part of me, on the one hand, would actually like useful systems, even if I don't understand them. If something is useful enough that it actually allows me to predict the world and do something useful that benefits humanity, one part of me called that the engineer part of me or the utility useful part of me is satisfied if I can build useful systems. The scientist in me is not, because I think that I'd like to really understand what's going on just because it's useful isn't quite enough for me. And so part of the motivation, I think, for many scientists is in fact to really try to understand what's going on.

SPEAKER_01 38:50

I think engineers have already answered this question, right? But a lot of our technology is based on quantum physics um calculations and formulas that we really don't truly understand. But they're like powering a lot of like the lasers and the satellites and all the stuff that's wrong.

SPEAKER_02 39:04

It's back to the weather predictions. Right.

SPEAKER_01 39:07

Yeah. And we don't understand them, but we're using them all the time. And so, of course, you can use something that you don't understand and still try to seek to understand it.

SPEAKER_00 39:16

No, but my point was not either of those things. It was given that that we've lived with this dichotomy for a very, very long time, even in a non sort of native digital way. Why is it that the general public cares so much about security and safety and guardrails? We talk about this all the time. I'm saying, would you give give up the security for the sake of the discovery? Because that to you is more compelling and exciting and advantageous to human beings. You don't mind not understanding it because hopefully we'll catch up. Let's just let it run. Or do you think no, we we need those guardrails, a Stuart Russell kind of mindset?

SPEAKER_02 39:58

Yeah, I I I think this is one of the tensions. I think in the context of science and scientific discovery, uh, I really would try to encourage discovery in the sense that we know that technology advancement and scientific discovery has led to millennia of humanity's progress, whether it's dealing with diseases, understanding the world, the way the world works, inventing and building the world around us, uh, everything we're doing now is a result of scientific and technical discovery. So in that sense, more discovery, please. Uh more discovery, please. I think there's a separate question when we think about putting uh technology

Interpretability And Explaining Latent Spaces

SPEAKER_02 40:40

like this into society for other uses beyond discovery. So I think in that context, then I think it's important to think about the the benefits and the complications, the benefits and the risks. Trevor Burrus, Jr.

SPEAKER_00 40:55

Of like open source models or whether Yeah, or any number of systems.

SPEAKER_02 40:58

I think the the risks come from not just open source systems, but they come from any number of things. They can come from even people using closed systems to do cyber hacking. Malicious things. Yeah, bad acts and fakes.

SPEAKER_03 41:12

Yeah, yeah.

SPEAKER_02 41:13

So the risks aren't just the ones from open source systems. So I think when you when you put these things into society broadly, then the question of the right balance between benefits and utility and risks, I think becomes so important. But if we're in the realm of science and discovery, I want more of it. I think it pro it's progressed humanity. It's led to breakthroughs. In December last year, one of the Nobel laureates, Joel Moike, in his Nobel lecture, talked a lot about he's an economic historian, which I thought was fascinating that an economic historian actually got a Nobel Prize, because he talked about just the history of human progress and how it's always been this interplay between scientific discovery and technological progress, the interplay and the bi-directional nature of that interplay, where science has advanced technology, technology and the tools that we technically build have led to more discovery. That that interplay between those two things have led to human progress, societal welfare. So, in that sense, I want more of that.

SPEAKER_01 42:21

I think you referred to this as the enlightenment and industrial revolution at the same time.

SPEAKER_02 42:25

Absolutely. And I think what's fascinating about that is the idea that, you know, AI may be the first uh technology we've had that has two characteristics that are unique. One is that it's a general purpose technology. Yes, we've had some general purpose technologies before, the steam engine, electricity, you could argue, is a general general purpose technology, general in the sense that you can imagine most human endeavors using those technologies, multiple sectors, multiple activities, et cetera. But it also has a second characteristic which is interesting, which electricity didn't have, which is this idea of an invention for making inventions. So when I think about the impact of AI in the best sense on human society, I think we almost want to combine, we often make the only just the Industrial Revolution comparison. I'd say I want to combine the Industrial Revolution with the Enlightenment, because the Enlightenment gave us ways of thinking, ways of inventing things. It gave us instruments, it gave us microscopes, telescopes, it gave us a scientific method. These are things that became methods of inventing. Either they gave us a way to think about the world, the scientific method, they gave us a way to look at the world, telescopes and microscopes, and as a result of that, we invented other things. So i that's why I like this combination of the Industrial Revolution and the Enlightenment in the best sense. Now, having said that, most of these benefits and possibilities. Also require us that we are thoughtful about a whole bunch of complexities and risks that come with that. And a lot of those are the ones you're referring to, Natasha, which is when we put these technologies into society, how do we think therefore about use, misuse, guardrails, and so forth? Because even with the scientific advances, we still need to worry about misuse in the form of biorisks, for example. Sure. People inventing viruses with these tools. So even there you've got misuse risks. So the kinds of risks that I worry about tend to be of at least a few kinds of any of these technologies. On the one hand, you worry about misapplication, misuse, whether in the scientific realm or in the societal realm, in the sense of, I don't know, deep fakes and those kinds of things. Then you also worry about alignment risks, which we were talking about before. So both things are true. And I I often worry that uh we too often focus on one and not the other, uh, depending which one you pick, the benefits or the complexities. I think both are true. And we should be solving for both maximally getting the benefits from this technology, but also taking seriously the risks and the complexities.

SPEAKER_00 45:21

I just think if you are truly using AI to explore unencumbered by current theories and biases and all of those things, and you really are going to the next level to try and make some breakthroughs in our understanding of why we're here or you know, a whole host of unanswered questions that we have. You just can't have both. You you're not going to be able to have these so-called guardrails. I mean, what would they even be? It's a bit like your other example of, you know, you set in stone something from 1957 and it's no longer fit for purpose. Obviously, uh, you know, morality's like fashion, it it changes with each generation. So I just

Guardrails Norms And Misuse Risks

SPEAKER_00 46:05

don't see how this tension can be resolved.

SPEAKER_01 46:07

But you can contain the system and have uh hand on the plug, right? So it's not that it has to be open in society running rampant and having access to the system.

SPEAKER_00 46:14

Well, I just mean even before you put it into society. I mean even at the um very early stages, as you've just said, a a bad actor could possibly create a virus in a lab when they were actually meant to be trying to find, you know, a cure for something. I I just nuclear codes.

SPEAKER_01 46:32

Yeah. Um somebody in a level five lab, right?

SPEAKER_02 46:35

Yeah. But but I but I think this is this is something, you know, I I'm a cautious optimist in the sense that I trust human ingenuity to work this out. I don't think you should say, on the one hand, everybody hear all these extraordinary tools, go explore unfettered everybody on the planet. We don't do that with most things. Uh we don't do that with most things. We've also historically been quite good at establishing norms for ourselves, even as a scientific community. I think, for example, back to uh the famous the Silama conference in the 70s when uh scientists could see where we're going with understanding the human genome and genetics and all kinds of things before it actually happened and set norms about what was appropriate to do or was not appropriate to do. Now, does every person on earth uh adhere to that at 100%? Probably not, because we know there have been some instances of the kind of cloning we would look down on. But for the most part, as the scientific community, in a very large measure stayed within those norms. Yes.

SPEAKER_01 47:46

So I think you would have to re- But even the Chinese uh locked up the guy who who did it. What was his name? There was this one person, right?

SPEAKER_00 47:52

So I think you have to rely on some version of the But that's a different that's already a different era. As you you talk about the compression between the question and the answer.

SPEAKER_02 48:01

Right.

SPEAKER_00 48:02

You know, where we're living right now, these things are almost instantaneous.

SPEAKER_02 48:05

So True. But but but I think I'd I'd rather rely on us evolving those kinds of methods than because what's the alternative? The alternative to say stop all the time.

SPEAKER_00 48:15

But I wonder if AI, but no no, actually the opposite. I I wonder if, you know, whatever we've been doing to nature for millennia, you know, AI may do to us. And um or maybe not. It may treat us much, much more kindly. I I guess, you know, it's a radical thought, but I'm just interested in whether there's anyone in your community that doesn't have a reputation for being uh the accelerationists.

SPEAKER_01 48:40

Yeah, they're all like settled to the metal, right? Yeah.

SPEAKER_02 48:42

No, the the the there are people who think of themselves as accelerationists, in other words, but uh, you know, my my own view and and thankfully, you know, many people that I work with is the idea that we we have to both be bold and responsible.

SPEAKER_00 48:56

Uh for as long as possible until you can't. Yeah.

SPEAKER_02 48:59

Yeah, I think we have to. And I think we have to, as a community, take both things pretty seriously. And I worry when everybody comes down on one side of that and not the other. I think both things are true. So I take, you know, I'm very proud of the fact that many of us spending an enormous amount of time doing fundamental research and alignment on safety and security. How do we red team these systems? How do we track the progress? How do we monitor and evaluate whether they're getting to, you know, uh dangerous levels of capability? How do we keep monitoring that and evaluating that? So I think as much effort should go into that, as much as the effort we put into developing useful uses for of this technology for humanity. I mean, I think there's just so much uh, you know, I'll I'll give you an example. I spent some time uh uh for a couple of years, I was kind of uh I was the co-chair of the UN's high-level body on AI. And when you look around the world, uh, and in some communities and and in some countries, people have don't have access to doctors, don't have access to schools. Uh and in fact, I remember being struck by the fact that when we were talking about the risks of AI in the high-level body, uh, we talked a lot about misapplication and misuse. But then many people from the global south and other communities said, no, no, no, there's another risk we should add, missed use.

SPEAKER_01 50:33

I love that term.

SPEAKER_02 50:34

We we're in places where what's the alternative compared to what? We don't have doctors, we don't have access to this, that, and the other. If this technology gives us the possibility of having access to those things, access to information, access to medical advice, you know, unless somehow the world magically helps us get to those things, this technology could be useful. So I think it's those kinds of uh uses of technology in the public interest and to help progress society everywhere that I think is what makes this worthwhile. Think about the applications, and there's a lot of this in the volume, by the way, in the Daedlass volume, in healthcare, in the health sciences, uh in understanding diseases. It is already the case, and there's a wonderful essay by Eric Toppel, who's one of the most scient cited uh biomedical researchers in the world, by the way. I think is, you know, he's he's written so much. But there's now enough studies and examples that showed that, for example, uh medical practitioners assisted by this technology do much better.

Missed Use And Global Access Gaps

SPEAKER_02 51:40

Much better than practitioners not assisted, especially in diagnosis.

SPEAKER_00 51:45

I think that's hugely celebrated and and known, isn't it? It's I I I'm I don't think anyone's disputing that. What surprises me though is you can only get to that place if you've had some unbridling at some point and some ability to be Wild West about it and make discoveries. I mean that's how we've always made discoveries, isn't it? Sometimes by accident, sometimes by going for one thing and then ending up discovering something else. The other question I wanted to ask you is given that that seems to be the biggest kickback about AI is the safety. That's what I hear all the time. So how much is being invested, you know, proportionately in that compared to innovation and pushing the You'd be surprised at how much work goes into safety.

SPEAKER_02 52:33

Because, so for example, often the safety questions themselves are fundamentally interesting research questions. So, for example, I was describing earlier the questions around mechanistic interpretability. It's actually a fascinating question. It's no different than uh cognitive scientists who are trying to understand how the brain works. Right.

SPEAKER_00 52:55

So it's still about capability in some level. Yeah. Yeah, it's about capability.

SPEAKER_02 52:58

I mean, we you know, we do MRIs, we try to understand which parts of the brain get activated when you are happy, when you're sad, when you're telling the truth, when you're lying. You know, that kind of research is also has been happening recently in AI systems. We want to understand when they make an accurate prediction, is there a part of the neural network that's activated, which one? Uh, we now have these systems that are these so you know, so-called mixture of experts architectures, where we know some part of the AI architecture is doing speech, especially in in natively multimodal systems, is doing speech, some parts is doing reasoning, some parts. So we're curious. So in on our way to doing uh tackling some of these safety guardrailed questions, there's some fascinating science questions that computer scientists in this case are almost like cognitive scientists trying to understand what's going on. Uh, we've spent an enormous uh amount of time trying to think about how how would you even watermark these systems? Uh, the watermarking question is an incredible computer science problem. Uh, so in our case, we've ended up being able to, for example, create something called synth ID. Uh, and I can tell you the people who worked on that were just so thrilled about the breakthrough because that's a fascinating question about how do you do that? Think about the people who are working on safety in the sense of uh securing uh cryptographic systems. Uh the mathematics and science of cryptography, particularly with quantum and quantum AI systems, these are fascinating problems. Uh so I I think I think the, you know, it's not as if the safety-related research is boring and uninteresting.

SPEAKER_00 54:43

That's good. No, thanks for clarifying that.

SPEAKER_02 54:44

It's actually quite, it's actually quite, but I think we always need more of it. Uh we always need more of it. I think as a community, and and and I I guess the part uh, you know, to comment on your Wild, Wild West comment earlier. When it comes to scientific discovery, I don't think it's as much a Wild, Wild West as people think. So one of the things you see in the volume, uh, and and you see it in the scientific journals, a lot of the AI for science work is being played out in scientific journals. They, you know, the the work gets a lot of peer review. They're scientific trials. For example, we're doing a nationwide trial uh in the US to try to understand, for example, AI assistive tools for physicians. So it hasn't been thrown out into the world. It's going through trials and there are

AI Changes Science Into Validation Work

SPEAKER_02 55:35

regulations that guide those things. You can't just put AI assists into a doctor's office.

SPEAKER_00 55:40

Uh so I think society Well, people already use their Chat GPT as their doctor anyway. So this that that's been crossed that threshold.

SPEAKER_02 55:48

Well, for individuals, yes. I'm talking about consistent tools for clinicians in hospital settings. Uh I think the questions around what do individuals as consumers want to do, I think those are questions for democracies to work out. I think societies and democracies should work those things out and decide collectively what they want. Uh, I think that's not a question for technologists to decide. I think society should decide that communities and citizens should decide those questions.

SPEAKER_01 56:20

I think this uh is also something that really came out of your paper to kind of go back to it as we close. The reality also for scientists is that we now are on a precipice where a lot of the work is actually asking the right questions. It's it's no longer really just doing the research and finding things. And and I wonder if we're teaching people how to ask the right questions and whether we have to change the way we approach all that.

SPEAKER_02 56:46

Yeah, no, I I I I think, I mean, you're you're right. I mean, one of the things that certainly that I try to point out in the in the essay, and I think it is the case in the field, I think the impact of AI and science, in some ways, is changing what it means to do science. Uh, because, you know, it's changing, for example, uh, I mentioned this kind of ontological inversion that I was talking about in the volume, which is, you know, in some ways, the it used to be the primacy was all around what you do in the lab and how you validate things. And it was actually quite often quite hard uh to come up with ideas and conjectures. Now, in this case, these systems come up with a lot of them. That's for example, in material science, one of our systems came up with something like 2.2 million theoretical crystal structures that had never existed before. Now, the vast majority of those are not stable. In fact, when we went to validation of the 2.2 million, there was something like uh a couple hundred thousands that were stable enough to be synthesized. And even then, the ones that are being synthesized are more in the tens of thousands as opposed to so, in other words, you have this vast space opened up by these systems. Uh, you still have to figure out which of these match reality. And what's interesting about that is that in some of those, there's real novelty. Uh, another example that I actually uh there's a paper that I quote in the in the volume where we're uh an AI system is exploring vast territory around potential cancer treatment pathways. It came up with something like 36 uh pathways. Half of those were already known, uh, so not that interesting because doctors had figured it out some other way. Uh, another, you know, 12 or 10 or 12 of those were just total nonsense. Then there were another five that were kind of oh sort of interesting. And among those, one turned out to be actually truly novel, a truly novel discovery. But it took discerning scientists figuring out what was already known, what was nonsense, to find that novel idea that they hadn't known before. I think you want that. I think you want it.

SPEAKER_00 59:08

It's still worth it, right? It's worth it.

SPEAKER_02 59:11

And that one example is actually validated in a lab at Yale. Uh, that, oh, this is actually a novel pathway for something. So it changes the question often to maybe the challenge becomes how do you validate ideas as opposed to come up with ideas? It's much more about validation. It also changes the picture from you know, exploration, because before most scientific experiments started out with the hypothesis. So you're always constrained by what hypotheses you have, what theoretical

Who Benefits And Closing Reflections

SPEAKER_02 59:45

frameworks you had, uh, were kind of almost the guardrails, if you like, of the range of theories and ideas you would have. It's a bit like the goal player. If you had asked the goal player, what are some hypotheses about what strategies would win? They'd just be narrowly confined to the history of human experience playing Go. But now you can with these systems allow us to go much beyond that. Now, is there going to be some hallucinations in that? Yes. Are there going to be some nonsense in that? Absolutely. Are there likely to be some breakthrough innovations within that? Yes, there will be. And I think that's what's exciting. The possibilities are just that's what's very exciting.

SPEAKER_01 1:00:28

Well, that's uh a beautiful place to wrap this up. Thank you so much, James, for spending the time with us and uh giving us a view into where AI is taking us on the scientific world. I think you already wrote an addendum to this uh paper that you wrote in January. So hopefully by the next year in January you can come back and tell us about the addendum to the addendum to the addendum.

SPEAKER_02 1:00:50

The uh the American Academy of Arts and Sciences has only done kind of this was the third volume of Debus focused on AI. The first one was in 1988. Then the next one was in 2022, and this was in 2026. So I don't know how much longer we'll wait for the next one, but um but they've all been asking very different questions. Right. Uh which is what's fascinating.

SPEAKER_00 1:01:14

Yeah, we didn't even get into culture.

SPEAKER_02 1:01:16

The other thing we didn't get into, which, you know, you mentioned Kelly Chibala's paper uh writing about AI in Africa. I think one of the key questions that's worth keeping in mind is uh history has shown us that even when we make breakthrough discoveries in science, it isn't always the case that everybody benefits from them. And I think that's one of the things we'll need to make sure happens. Because we know, for example, what happened in COVID, right? We the world discovered and invented vaccines. Uh for a while, a few countries hoarded them. So I think the question just because we make scientific progress and even discoveries, doesn't necessarily mean everybody benefits from them. So I think that's one of the things we'll need to make sure, which is as we make these discoveries and these breakthroughs, how do we make sure everybody benefits? That scientists everywhere can make use of them, that uh society as a whole benefits from them. That problem won't solve itself. We have to actually work to solve that. So I don't just worry uh about Natasha about the guardrails. Of course, those are very important. Safety is important, or how we bring this technology to society is very important. But even on the positive side, making sure that the extraordinary progress and breakthroughs and benefits are, you know, everybody gets access to them and participates in them is also just as important. Absolutely.

SPEAKER_00 1:02:36

Yeah, of course.

SPEAKER_01 1:02:38

I think we have a good guardian of those values there. Thanks so much for doing your work, James, and appreciate you taking the time. Thank you for having me.

SPEAKER_00 1:02:46

Thanks so much. Thank you.