Podcasts›Automated›Dhruv Batra on what AI still cannot do in the physical worldLike💬 Be the first to comment →AudioSeptember 15, 2026Dhruv Batra on what AI still cannot do in the physical worldFrom Automated, hosted by Brian HeaterYutori's chief scientist on the gap between a model that can talk about the world and a robot that can act in it.Play episodeSaveOpen the audio file TranscriptTranscript from the show's feedSearch the transcript0:00LLMs are, you know, fantastic at language, dialogue, reasoning. However, crossing that bridge from the digital world to the physical world is still challenging. What we are really seeing is a glorified physical wrapper around ChatGPT. I wanted to find a way for intelligent systems, when faced with uncertainty, to give back multiple plausible responses. When I realized that chatbots were somewhat underwhelming because they didn't do anything in the world, you just get a response back.0:31I wanted actions to be taken, and that's what took me to robotics. Yutori builds web agents. These are agents that take actions on a browser. I personally think of them as robots of the web. We never build a map. There is no explicit planning module, there is no collision sensor, there is no LIDAR, there is simply a neural network that goes pixels to actions. I'm wondering if, in terms of seeing the bigger picture, whether people in the machine learning and AI find it useful to have these philosophical conversations or these conversations with like sociologists, representatives of the humanities from you know outside of your specific field?1:14I certainly think it happens. welcome to another episode of Automated. I'm your host, Brian Heater, the managing editor at the Association for Advancing Automation. We've got a fun one today with Drew Batra of Yotori.1:44It was it was a strange week for me, so I started off with a big philosophical question and things kind of unraveled from there. honestly, really enjoyed this conversation, and I think you will as well. Thanks so much to Drew and your Tori. Thanks to you for tuning in. If you're enjoying the show, you can help us out by liking and subscribing, checking out the newsletter over at automated.fm. In the meantime, please enjoy this conversation with Drew Batra. Vision and AI are changing what's possible in industrial automation from quality inspection and predictive maintenance to robotics, logistics, medical devices, aerospace, and manufacturing.2:24At the Advanced Vision and AI conference, you'll hear from industry leaders, see real-world AI applications in action, explore the latest in machine vision, computer vision, sensors, edge computing, and automation technology. You'll also be able to connect with the people who are shaping what comes next. Whether you're building, buying, or implementing these systems, this is a chance to stay ahead of the curve and see how vision and AI are moving from promise to practical impact. Learn more and make sure to buy your ticket early at the Advanced Vision and AI conference September 23rd and 24th in Santa Clara.3:01I'm going to read from your Georgia Tech bio, but I thought this was really interesting. And I assume that you had a hand in writing some of this or co-authoring it. his research interests lie at the intersection of machine learning, computer vision, natural language processing, and AI with the focus on developing intelligence systems that are able to concisely summarize their beliefs about the world with diverse predictions, integrate information, and beliefs across different subcomponents or modules. Okay. The part that I want to pull out there is intelligent systems that are able to concisely summarize their beliefs about the world.3:39What does it mean for an AI system to have a belief about the world? Like what is the analog there? Oh wow. What a fascinating place to start. and it's a it's a blast from the past because that line is actually coming from a body of work I did 13 years ago at this point. and over time, what I mean and what the field has meant about summarizing beliefs has evolved.4:10So to step back when I So it's a good place to start then, so we can work forward from here. Yeah. circa 13 years ago, I was a young assistant professor at Virginia Tech, and I was embarking on my my independent research career, and what I wanted to do was find a way for intelligent systems, when faced with uncertainty, to give back multiple plausible responses. and it was important for me for intelligent systems to convey their epistemics properly, that you know, you may not know the answer because either the problem is ambiguous, or you are not particularly sophisticated and capable, or you know, the solution just admits the problem just admits multiple solutions.5:03And so this may be up to a preference. And so what I was trying to do was find algorithms that could not just present one answer to a question, one path in a search problem, one solution, but multiple plausible solutions. and in some sense, we've already developed the mathematical machinery for conveying beliefs. We that you know goes back 100 plus years to the idea of Bayesian probability.5:34Since you're starting from a philosophical place, you know, my intro to machine learning class started day one with a slide of what is probability? what does it mean to say an event has a certain probability? And there were three interpretations. first, a frequentist interpretation, which is probability is a limiting ratio of frequency of an of an event. if I could repeat an experiment and infer the number of times within those confines that it would recur. Number two, it could be a subjective belief.6:06It could be a belief that an observer or an or a system or an intelligent system holds about that event. and then that belief just has to abide by rules of rational belief updates, Bayesian updates. number three, which most people today are familiar with, but back then were not, is that a probability of an event is what a rational buyer is willing to pay in a prediction market for an event contract. If it were this is the Calci answer to the question.6:38This is the Calci answer, except this class was taught in 2013. Calci did not exist exist. Prediction markets as a concept, as a mathematical concept existed. And and so, in some sense, we've had this answer to what does it mean to believe something for a while? Intelligent systems try to build on that machinery by saying, okay, if I can, if you ask me a question, what is the capital of France? You know, I my beliefs should be fairly peaky. you know, it's a factual question, there's very little ambiguity.7:09But if you ask me a question, you know, what do we understand about the fastest way of reaching to an exoplanet? Well, there's so much that we don't understand, there are maybe some best practices, and so I should give you multiple plausible answers. And so that's where that line is coming from, of trying to build intelligent systems that can give you that can cover the space of their beliefs by giving you modes in the distribution, if you will.7:41Your your belief space has multiple bumpy heights where where that correspond to solutions. and how do we extract that? and so you know, among other things, my group came up with an algorithm called diverse beam search, which is an algorithm for sequence models from decoding multiple responses that are diverse and different. At that point, this was research being conducted in 2013 to 2016 period. This is pre-LLMs, this is pre-chatbots.8:12And so we were still thinking about probabilistic systems that could be used in computer vision, robotics, NLP. over the years, now in the post-LLM era, I think when people talk about beliefs, they almost collapse into how does a language model express uncertainty about questions that it has asked? And in some sense, the answers to those questions tend to be, what have you told it in the training data to respond?8:42Which is a different place to start. Yeah, okay. So the good news is this was a good first question. The bad news is there's a reasonable chance that we might be stuck on this the entire time, which I'm I'm perfectly fine with. so let let's take because I I think the capital of France is a good example, right? So let's say let's give two examples of that. Let's give the example where you know you you you know for a fact that the capital of France is Paris, right? Let let's l and then as as the second example is you think it's either Paris or Toulouse, right?9:20Both of those would qualify as beliefs, even though one is like relatively objectively the right answer. Yeah, so this is where we would get into an epistemic uncertainty versus alleoteric uncertainty. the distinction between an observer's belief about the world, you and you an observer is allowed to start without any a priori knowledge about the world. And so therefore you can start with you know not knowing much about the world, maybe you've interacted a little bit with the world and that updates your beliefs.9:54And so over the course of your lifetime, maybe you just haven't read you know sufficient background information. And so you can believe with nearly equal probability that the capital of France is of France is Paris and Toulouse, and so in your belief space where on the x-axis, let's say some ordering of cities on the y-axis represents the strength of your belief in them, and maybe there are two peaky modes in that distribution that can exist for you, and that's totally fine.10:25The only thing that the Bayesian interpretation of probability enforces is that when faced with new information, there are rules of update that a rational believer must follow. And you know, as long as you are doing that, your you know, your beliefs are perfectly valid beliefs. Those are those are mathematically consistent objects. and therefore interaction with the world ends up being important. When you interact with the world, you realize that other people have different beliefs.10:58And when you actually land in Paris and you see a sign that actually mentions it's the capital of that country, you had better update your beliefs because that's a pretty strong information coming from the world about it. So yes. Well, okay. Yeah. I mean, talk about like decision making. Again, there's like five or six directions I want to go with in this. So let me pick what I what I think is is the right one, or at least for boiling this down to fairly simple language. You know, you talk about this openness to to accepting new information, which, you know, and and and that's again, you know, although LLMs are like obviously like these gigantic like black boxes, you know, assume that like all the LLMs we're dealing with right now are trained on the fact that Paris is France, right?11:47That information is probably in every LLM that you're using right now. but we also know that Versailles used to be the capital of France. So is that part of the reason why you need to leave an openness there? Is the fact that the information itself could is dynamic? Or the phrasing of a question coming to you may have associated context that is missing in the question conveyed to you. and so one of the one of the key tenets or a key implication of Bayesian decision making is that a rational decision maker cannot assign zero probability to exactly zero probability to an event.12:34Because once you assign zero probability to an event, you will not update it because it's a special case if you work through the, you know, this is elementary, high school, early undergraduate algebra. But if you work through Bayes' rule, once you start with a belief that is assigning zero probability to an event, no amount of incoming information will move you from that zero. That's why you have to, in order to keep an open mind, it just needs to be epsilon away from zero. You're head- you're hedging a little bit. You're hedging a little bit.13:05And that's leaving you open to new information coming in. I'm I'm I'm stuck on the word belief from somebody who, you know, I was not particularly good at undergrad algebra. I was a humanities major, I was a creative writing major. so I'm very interested in the significance of the language that we use. And you know, we at least like abstractly have a fairly good idea of what belief means in human terms, right? And or like what a deeply held belief is, what like spiritual belief is.13:37And I'm wondering to what extent is it helpful to use these human analogs to describe artificial intelligence? It's a wonderful question. I've I've thought about this from both perspectives. so on the on the one hand, a lot of times what we engage in and the AI in particular gets criticized for using what are known as suitcase words, where we introduce a term where there is a colloquial sense of that term, but AI researchers are using it in a very specific way, or worse yet, they're using it in a multitude of specific ways.14:22And they're like we can be accused of playing almost a bait and switch where we are we are we are phrasing a statement. So when we say LLMs believe something or intelligent systems believe something, what we mean is a very specific thing. Yeah. What is a belief? It's a it's a probability distribution. What is a probability distribution? It is a mathematical object that that follows certain axioms of probability. what does it have to do about the world?14:52Well, it makes certain predictions, we can incorporate data from the world, we have rules of updating beliefs. That's what we mean. But there's a colloquial sense of belief that comes along for the ride. and this leads to misinterpretation of mathematical results or you know, scientific results. Let me let me like posit a an analogy that I think probably will resonate with our our viewers and listeners, and one that you can relate to, especially like given some of your background with with meta and some of the work that you've done.15:23One of the big things that I'm running up against as somebody who reports on robotics is the attributes that we imbue a humanoid robot with, what it can and can't do just based purely on the form factor, which I I like that's a pretty fair analogy, right? Yeah, it's a fair analogy. when we look at another entity, we attribute capabilities that are correlated with the capabilities we have seen in other entities that look like that.15:57And so roboticists are deeply aware of this. and this is why, or at least pre-LLM era, there was a very sharp aversion in robotics literature to allow robots to speak because the moment you introduce language into the picture, the amount of sophistication expected by users of robots goes through the roof. If I can talk to it, then I expect it to understand what I'm saying and I expect it to do the kinds of things that I would expect and another entity that I can talk to to do.16:33This is one of the reasons why LLMs were such a breakthrough because it was like finally delivering to a certain extent on that promise of being like dynamic in that way. Yes. however, there is still a problem. LLMs are you know fantastic at language, dialogue, reasoning. the digital world is their conquest. Crossing that bridge from the digital world to the physical world is still challenging. You can talk, you can, you know, in the last couple of years, especially post ChatGPT, there were a bunch of robotics videos that people put out where what we are really seeing is a glorified physical wrapper around Chat GPT.17:15Like you put a robot on you put a robot on a table and the person is having a wonderful communication with the robot, and you know, API calls are being made to GPT behind the screens. and I've I've looked at that you know videos like that and gone, okay, we know how to, we know how this is done. This is ChatGPT. What difference did it make that there was a there was a physical robot here? Show me dexterity, show me navigation, show me physical understanding, show me you know manipulation in the presence of other entities where you have to think about their movement and and and make predictions about physical space.17:48To sort of back up to the question that you were asking, when we when we use words that have colloquial meanings, we do run into this problem that sometimes the intent is misinterpreted, and and that is a there is a criticism and then an illegitimate criticism. Having said that, I will defend it a bit in the following way. In some sense, the job of science, engineering, knowledge building, mathematics, if you will, is to make precise vague colloquial notions that we may hold in our head and make them precise in a in a rigorous way.18:34So to to distill them to their essence effectively. Exactly, to distill them to their essence or at least show an instantiation. and so the I and I'll give you the most recent instantiation of this example. Today, when you talk to an AI researcher Sotly, but now increasingly users of chatbot, you can use the word the chatbot is thinking. And you know, we have in some sense systematized thinking.19:06What does thinking mean now? Thinking means rolling out an internal monologue in terms of tokens before you produce a final response that is emitted to another party. That is what thinking means now. It is a pretty narrow definition of thinking. I'm sure, you know, that's not how neuroscientists think of thinking. that's not that's not how cognitive scientists think of thinking, that's certainly not how philosophers might agree on thinking. But we've sort of even in AI, we've sort of converged on an extremely narrow definition.19:40Thinking is a sequence of tokens that are unrolled from a language model before an output set of tokens are produced. what have we lost in that? Well, we've lost the richness of what thinking could possibly be. But what have we gained from it? We've gained a certain precision. We we can talk about what this means. We can talk about a thinking budget, we can talk about the impact of performance with thinking on the x-axis. You couldn't do that with a vague notion. and so, yes, it's a trade-off, but I see both sides of that trade-off.20:13This is this is fascinating too, because you know, I'd be curious to see some, you know, especially I've actually spoken to a few people who who like did dual majors in you know electrical engineering and philosophy. And I think they're uniquely qualified to tackle some of this, but to kind of reverse engineer these human concepts from what we've distilled them to AI, if that makes sense, from the standpoint of when you talk about tokens, you're talking about a reward-based system, right?20:47I mean, and and you know, whether whether there is an analog, you know, in in in humans, like if if if there is like effectively something that we could use, like some version of a token that drives our thinking. Well, tokens are particularly odd because they're quantized subword units, and in most modern LLMs, they're byte pairs. There's almost yeah, it's extremely unlikely that we or you know nearly impossible. I'm I'm struggling because being a rational Bayesian, I cannot put a zero probability, but but it's extremely unlikely that you're doing it.21:24You're you are doing it though, because you are like, you know, again, this is this has rewired your brain in a certain way wherein you are now thinking of some of these philosophical humanoid humanoid. Wow. That see that that that was a version of the two. Yeah, exactly. for you and flip. that but it has recontextualized the way you've been thinking of of the way you think. Yeah. I mean, I mean, this is the classic Kunz paradigm, right?21:57When a new paradigm comes on the scene, it It provides you a perspective. The hot new paradigm. Yeah, it it shapes the the it shapes the guardrails, it shapes the the questions you're allowed to ask and you're not allowed to ask. And particularly early researchers don't even realize that there could be another paradigm. I'm old enough that I went to grad school studying AI, not just pre-LLMs, but pre-deep learning. So, you know, I've seen the transition from we will build probabilistic systems that reason about uncertainty to, you know, don't worry about multimodal high-dimensional probability, just you know, get large amounts of data, we will train massive neural networks, to we will just build the foundation of intelligence with large language models.22:45Each one of those paradigms has you know has benefits in the sense that it it we've been steadily making progress, but it shapes the questions you're allowed to ask. and so yes, I I agree with you that what a paradigm does is it you know rewires how you think about the world. but hopefully, and then the interesting thing about AI is that we've had so many paradigm changes so quickly that it there are going to be a growing set of people in the world who can see past paradigms and who can see the the holes in the current current set of paradigms.23:25Oh, interesting. And and then there's like a broader connective tissue like throughout all of these, perhaps, that we're maybe not seeing right now. Yeah, I mean, it it it's I I think there is there's growing agreement that throughout the paradigm changes, there are some key ideas that we're settling on that are essentially the right ideas. that you know, post-the transformer architecture, the yeah, you know, we we didn't freeze the transformer architecture in place for over 10 years.23:56There have been changes to it, but the idea that associative attention-based memory is a key artificial neural substrate that we will use for learning is just you know, that's an idea that we think is so powerful that we we will not end up overturning it. The idea that you know, this goes back all the way to Turing, 1940 paper of his, where he mentions that instead of trying to build an intelligent system completely fully formed, why don't we learn an intelligent system?24:33Like he didn't use the word machine learning. Today what we call learning wasn't exactly what he was thinking about. But the idea that why don't we grow a baby intelligence into a fully flawed formed intelligence, like that we're not going to that is an essential enough idea that has been baked in. We're not going to approach AI by trying to build an expert system ever again, where you know learning is going to be the way we're going to approach intelligence. And now we're just shaping through, you know, if we're going to do learning through, it seems like first-order gradient-based methods with neural substrates that look like associative attention-based memory of our of transformers.25:14Now there are a whole bunch of problems to be solved around how do we, you know, continually learn on the fly without forgetting past information? How do we improve data efficiency? There's a whole bunch of questions that we have to have to resolve. But throughout these paradigm changes, I think we are settling on key fundamental ideas that aren't getting revoked across those transitions. I like the training example because that that's a great example of obviously like one of the brilliant minds of the 20th century.25:45And also I think what you're getting at there too is that he had really big picture thinking that he was able to think in sort of like abstract ways. Obviously, he couldn't think through all of like the exact things that we've spent the last 50 years working through, but he could see that bigger picture. And I'm wondering if, in terms of seeing the bigger picture, whether you and I I and so far as you're aware, other people in the machine learning and AI fields like find it useful to have these philosophical conversations or these conversations with like sociologists, you know, representatives of the humanities from you know outside of your specific field.26:31I certainly think it happens. you're you're catching us at a by us I mean the broader AI community at a particularly interesting time frame. and you know, I I can tell you what things looked like in 2005 to 2012 time frame, 2012 to you know 2022 decade, 2022 onwards. And you're catching us at a at a point today where progress is so fast that there doesn't seem to be a roadblock in sight that right now perhaps people do not have the mental space to look elsewhere for ideas.27:18You go talk to other people when what you are working on seems stuck. and right now the AI community is you don't have the you just don't have the bandwidth for it, is basically is what you're getting at. It's not stuck. I do find I personally have changed fields a number of times. you know, I I was classically trained in my PhD in machine learning and computer vision. So detecting objects and images and videos was a thing that I studied in my PhD.27:50After my PhD and my postdoc in my first faculty appointment, I drifted into vision and language, today what is known as multimodal AI. So help building chatbots that can answer a sequence of questions about images. And it was only a few years after that that I got interested in building intelligence systems that can actually take actions in the world, which is what took me to you know, first virtual robots and 3D simulators and then physical robots in robotics. That kind of meandering, I think, is somewhat rarer today and harder today because of the extremely high opportunity cost associated with it.28:31The opportunity cost is lower when progress is slower. and so I do think that people are less inclined to make rapid pivots or extremely sharp pivots in their careers because frankly, things are working. yeah, the this is another example too of why we can't go to zero probability when discussing when looking at the Turing example or other like big abstract examples, you know, you you you're implying that like okay, like we've we, and by we I mean you collectively have settled on this kind of larger model, but you still have to continue to be open to the fact that like there are going to be huge paradigm shifts that you're going to have to grapple with along the way.29:23Yeah, absolutely. And you know, this is the the beauty and the frustration of the development of new knowledge. you are almost certain that there will be new paradigms, but you cannot see past them because if you can't, there wouldn't be new paradigms. and so you just have to keep an open mind. and you have to you have to keep making progress in the in the direction that right now seems fruitful.29:54There's a there's a skill set that is that you develop in in a research career, and you know, as as PhD students or early researchers make their way through their career, there's a the skill set of being able to predict what problem will be important and ripe for solving some horizon ahead of you. Without without obviously without being able to predict the answer, but just knowing what questions to ask. What questions to ask? and it and people go through this this transition early in their career where you know if you don't have any research background and you're working on your first research projects, you probably can't even see the solution of that project, let alone any question that comes beyond it.30:39Because everything is new. But there are often these these academic stories of mathematics PhD students clearing out and their, you know, maybe their advisor is moving offices and you find a diary with a solution to their PhD thesis problem written at the back of it. Like they knew it all along. Just like scratched out, scratched out, yeah. They just needed you to grind through it for five years. but but whether or not the the you know, leaving aside the veracity of those stories, but I think it's generally true that you're when you're when you're new to research, the person who's advising you can probably see the contours of the solution much clearer than you can.31:19But also you gain the skill of being able to predict what questions will be asked at the end of that research, what questions will be important six months from now. by the time you've been in the field a few years, I think most researchers are able to predict the question that a bunch of other researchers will be asking next year. or you know, 18 months out. So this is another thing I think that you're in an interesting place to answer. And I haven't gotten I've gotten answers, but I don't feel like I've gotten a satisfactory answer.31:51And maybe that's because like I want a certain answer out of it. but is as somebody who has worked on AI for so many different generations and has worked it you know in and around robotics as well, when something like Transformers come along, when something like an LLM comes along, completely changes the system. And let's say, like, I I know I I know robotics better, so I'm I'm just gonna use that analogy. But it it it it you know, we've been hard coding robots since robots existed, right?32:27And now this is like, okay, the promise to no longer have to hard code robots. And that was hours and hours that everybody spent doing all of that that that difficult hand coding. When when something transformative comes along and just completely shifts the paradigm, is there sort of a sense of regret that you feel internally that all those hours were spent doing it the hard way? Yeah, fascinating question. I I will I will channel roboticists, particularly perhaps the pedantic roboticists who will point point out that not all roboticists have been hard-coding robots.33:04Dean Dean Pon Pomelaru, and I apologize if I'm butching up his last name, at CMU in 1988, built a system called Alvin, which was training a self-driving car to drive with imitation learning on roads, drove across the country with human corrections and feedback along the way. But the idea of neural networks or learning-based systems to train robots is an old idea. It's just it didn't it didn't make as quick a progress because the amount of data available in robots in robotics is is is small compared to other domains.33:41And the amount of work you have to put in to gather that data is is significantly more. so with that caveat aside, I the rest of your question, I completely agree. It's yeah, but it's an important caveat because it's interesting, and and we can point to cases of this like throughout technology of instances where the technology itself hasn't caught up with the it's so far ahead of its time that the rest of the industry kind of needs to catch up in order for it to actually be feasible. And sometimes it's technology being so far ahead of the rest of the industry, sometimes it's the glimmers of the right idea, but certain ingredients missing.34:23Training a neural network to follow, to learn from human demonstrations and control a steering wheel of a car and drive on a on a road that has the right contours of the of the idea, but there's so much more that you need. That's just this is not enough. and that so much more did not arrive for another two, two and a half decades after that. but with with you, the the the basis of your question still stands that the sun the sunk cost fallacy. Yeah, the sunk cost fallacy. And you know, roboticists my favorite roboticists to to read is Rodney Brooks, and he just wrote this incredible sequence of papers.35:05I think it was around 1990 where he wrote this sequence of papers, Intelligence Without Representation, Intelligence Without Planning, in which he really excoriates the robotics community for having pre- He loves doing that. He loves excoriating things in people. He does, and they are so much fun to read. writing that the robotics community prematurely committed to an abstraction and a modularization of the problem, that robotics is about breaking it into components of perception, planning, actuation, that there are certain commitments that each such module must make to the other one, that those commitments then enforce a certain performance standards on each of those modules.35:53And now off you go on each of it just it compounds. It compounds. and his criticism in those the thing is that this abstraction and this modularization was A premature, B wrong, and it has led to just years of of perhaps what you would call some cost fallacy, where we've been investing into that over and over again. And that you know, you what what you should build on the other hand are integrated systems that are interacting with the world, learning with data on the fly.36:25And you know, he's proposed a few different ideas over the years. And today that is how robotic systems are built, not precisely with the with the with the algorithms that that Rot may have proposed, but the idea that we we are going to build a perception system that builds a full 3D map of the environment or a 4D map of the environment, and then we're going to do planning through that 4D map with certain horizon, and then we're going to keep updating those plans. That is not how they are built. I my team at at Fair at Meta had this project that we released in 2020, where we showed that we could train this was virtual robots in 3D simulations, so not yet physical robots in 2020.37:08But what we showed was we could we could put in, we could train a virtual robot to navigate purely pixels to actions. So it would just take a camera image, a single monocular camera image, not even not even static. Static, but it would take that, you know, once every some period of seconds. and it would be processed. this is before transformers completely upended everything, but it would be processed by a convolutional neural network to get a vector representation fed into a recurrent neural network in LSTM to unroll out a sequence of actions.37:44Those actions are either velocities or can be discrete actions like move the robot body forward, turn it left by some angle, turn it right by some angle. So it's connecting the dots, the interstitials between the frames. Exactly. and it can see multiple frames as it goes along. And we never build a map. the the robot never tries to reconstruct its 3D space. there is no explicit planning module. there is no collision sensor, there is no LIDAR, there is not even a decomposition between perception and actuation.38:23There is simply a neural network that goes pixels to actions. So that that's more in line with what we do, right? Versus like having to map out an area in in a in a sense. Well, that's a bold claim that I I can't Okay. No, you you I I I I mean that okay, let me let me let me narrow that down. I just mean from the standpoint of in order to if I'm gonna go to work in a warehouse, like I don't need that precise map of the warehouse to work there. There's certain inferences. That is that is a that is a much sharper claim.38:54It is fairly certain that when we m ask humans for judgments of 3D physical distances, they do not, they cannot, humans are not able to make or or articulate judgments of physical distances between objects, but humans will act in those environments without bumping into those objects, which tells you that we perhaps do not have as precise a 3D map implicit in our head that we can that we can access in that same way. And we're probably not even building. Nor do we necessarily need it.39:24Nor do we necessarily need it. my my colleague Jitandra Malik, faculty member at Berkeley, but spent some years at FAIR, we used to say maps are simultaneously over complete and under complete. they tell you things that you simply do not care about. maps are trying to tell you, you know, whether or not there is a one millimeter or a two millimeter distance between a table's edge and the wall is irrelevant to a robot that just has to decide whether it can pass between the table and the wall.40:02Though both those distances are too small. and simultaneously they are maps are underspecified in that there is large amounts of information that you need that no sensor can tell you because you can't see the backside of an object whose front side you're observing. The reason why you know there is a backside is because you believe something about the shape of that object and you complete it in your mind. and and so this data structure is it does not seem to be the data structure that that could serve as the foundation of intelligence or at least physical or embodied intelligence.40:45Anyway, yeah. Yeah, no, okay, so so I think I think maybe an interesting thing to do at this point would be to again sort of break down these different periods of your career. So like 2013 as a grad student, looking at 2018 Caliper. So I think you founded a startup at that point, then you go on to Meta and and now you're you founded UTORE. I'm not gonna ask you to break each one of those down, but like sort of largely speaking, can you kind of I don't know, maybe maybe maybe give us one example of kind of a breakthrough that made it necessary.41:23And beyond that, I'm I'm just curious. Let me start with this. I'm curious as we talk about the Rod example, how you can commit yourself to a certain paradigm while being open to the possibility of change to the extent that you're able to make that pivot. Yeah. wonderful question. minor factual corrections, 2013 to 16, I was a professor at Virginia Tech.41:582016 is when I joined the faculty of Georgia Tech and simultaneously was a research scientist at joint meta. I was at both Okay, so there's overlap. Yeah, and there I was at both institutions till 24, which is when I resigned from both institutions, and founded UTOD, which is where I'm currently at as a as a founder. To answer the the research pivots and and questions, I think that's that's exactly the what my my career is is sort of shaped up to be.42:32Early on I was studying, you know, at this at the top of this conversation where we started representing beliefs from intelligent systems and how do we extract multiple plausible because the the guiding principle, how do we extract multiple plausible outputs that illustrate the beliefs of an intelligent system? The thesis was that when you talk to a chatbot or when you talk to a computer vision, or when you get an answer from a computer vision system or when you see a robot operate in the world, if the that intelligent system were to hand you a mathematical object like a probability distribution, a user does not know what to do with it, even though a probability distribution is a perfectly mathematically well-formed articulation of its beliefs.43:17And in order to make it interpretable, we were extracting multiple plausible guesses that an intelligent system might wake. and that's so we we don't have the Rosetta stone to read it directly, and you're sort of translating it, but you're translating it in there's but there's multiple potential ways to translate it. Yeah, there are multiple ways to translate it. We this problem of extracting what does a neural network believe is still an open question.43:50This is this today that research community is called mechanistic interpretability. this is a piece of work that my colleagues and I did in 2017, which was Called GradCAM. It's one of the highest cited papers in this field, which is how do you understand why a neural network is saying what it's saying or why it made a prediction it made? In some sense, this is related to the question of beliefs. What does this neural network believe? But this one is less couched in probability.44:20It's more asking, what is the process of computations that this neural network went through, or what part of the input did it look at, or what could have been different about the input for the output to be different? Asking a counterfactual question. So this was a phase of of my career where I was studying understanding intelligent systems and extracting their beliefs in an interpretable manner. then I got interested in vision and language systems or multimodal systems.44:53And there again, I had to keep an open mind because right around then is when you know the technology was shifting it almost entirely to neural networks, and you you wouldn't do this on the previous version of of the technology in AI. and I got fascinated by chatbots, chatbots that can look at an image and answer a sequence of questions, hold a coherent dialogue, spend some time building the first data set and the task for visual dialogue and visual question answering with my colleagues.45:30Last year, that piece of work was recognized with an award for having sparked this entire subfield of vision and language, which was quite muzzle tough. Thank you, which was wonderful to see. today the you know, I find myself sometimes just taking a screenshot of my of my laptop. Like if I if there's a list of numbers, I'll I'll just take a screenshot of those list of numbers, paste it in a chatbot like ChatGPT, Cloud or Gemini, and just ask it to total. behind the scenes, there's OCR being done and then a tool call being made and then a response comes out.46:04And it's it's wonderful to see the arc that connecting pixels to language and giving answers is a solved problem in a narrow domain. and then finally, yeah, right around here is when I realized that chatbots were somewhat underwhelming because they didn't do anything in the world. and so I wanted to qualify that as underwhelming.46:35Yes. Today had I fully understood the the consequences, I I probably would have continued with chatbots. but it but it was somewhat underwhelming that you know you just get a response back. I wanted actions to be taken, and that's what took me to robotics. Umbodied embodied AI. Embodied AI and then robotics. the the interesting thing in crossing the divide from into robotics is that there is a sharp line that roboticists draw around their field.47:07Roboticists almost immediately have the first question when you present them a result, which is Did you touch a robot? If not, not interested. This is this is not a robotics result because I had results in simulation. I was showing them virtual robots in 3D spaces that could navigate without a map. and my robotics colleagues, both at Georgia Tech and Fair, looked at me politely and said, please let's talk after this is on a real robot. And so I would I wonder I wonder if like like especially the work, the simulation work that NVIDIA has been doing has kind of loosened that up a little bit.47:45I think so. I think simulation used to be a pretty dirty word in the robotics literature, and it's less and it's less so. and it it it comes with historical baggage. I think the robotics community is almost as a PTSD, having seen results in simulation over many years that have failed to generalize to the real world, to real robots. and today that leap has been made a few different times. today, the dominant paradigm, if you want to train locomotion systems, locomotion systems like quad-legged robots, Boston Dynamics Spot or the Unitary Robots, if you want to train them to walk, trot, jump over objects, or even humanoids if you want them to walk, the dominant paradigm there is symptorial transfer.48:32You train them in large amounts of simulation with varying terrains to balance and under dynamical conditions, and you generalize that too. There is just like I think the field has realized that hand-written controllers or human expert-written controllers for high, highly dexterous dynamical systems like legs just does not match the same performance. so that that's been nice to see. Symtorial Transfer has not had the same success, generalizing learning from camera images.49:09So raw camera images, neural networks still pick up on the fact that an image was generated in simulation versus versus a real image. And that that has continued to be a problem, which is why you see a large number of dexterous manipulation results. So whenever people want to teach robots to pick up objects, they will often do it, they will typically do it with demonstrations. So you see these videos of human will be wearing these sort of fake gloves or or or controllers, and they will it's it's like it's like when it's it's like when a condor gets orphaned and they have the little fake bird to feed to feed it effectively.49:53Exactly. And and you you generate large amounts of data that you know it looks like an end effector of of a of a manipulator. you have humans solve those tasks in diverse lighting conditions, diverse scenarios, and then you feed that data to neural networks that are trying to then run it on the robot so that you can control the robot as a consequence. So we have not solved the Semterial problem for RGB cameras and other high-dimensional sensors like that.50:25But I think I think particularly with the rise of with with video generation models, I am certainly optimistic that that's the pathway that will play out. So at one point you I think reasonably jokingly said, you know, if you had known what you know now, then you wouldn't have left chatbots. but I think maybe maybe a good way of really breaking down the the progress not only the progress you made, but the the the progress that AI as a whole has has made would be to take Caliper and to take you Tori.51:03Because in some ways I think maybe they were trying to do similar things, but the technology has just like completely transformed like what can actually be done. you know what what what is what is the difference there? And and when when now that we're talking about like agentic AI versus versus chatbots, like like what what can we actually do right now that like was just kind of a fantasy in 2018? Yeah. so youTory builds web agents, these are agents that take actions on a browser.51:39So think about filling out a form that you know there's a large number of back office workers whose job, a large percentage of his job consists of, you know, an email comes in with a PDF, you upload that into a C into a software system, or you know, you're clicking buttons on a browser to execute, to book something, to buy something. that is the kind of agents that we that we build. I personally think of them as robots of the web. and I think of this as a digital space.52:10And I apply some of the same ideas and intuitions that these robots of the web perceive their world. Their world is not 3D, their world is 2D. They perceive their world through a screenshot of the browser, and they produce an action. that and that action is not a velocity command that controls the body of the robot, but it's it's an action that tells them how to act on the on the web page. So, you know, click a button, type this entry into this box.52:41Yeah. So what so one of the things, and correct me if I'm misinterpreting this, but it sounds like one of the key things here is who was I talking? I was just talking to I don't know if you saw the Walden Robotics launch. it's Russ Tedrick just like took his company outside. And I I was talking to their chief product officer earlier today, and we were discussing ways of training robots, right? And and and you know, there are a certain contingent of come customers who really want that natural language.53:17And to that seems to be a big part of what you're talking about here is is creating machines that can really like effectively execute on what you're asking them without having to do any hard coding or anything like that. Exactly. Exactly. No, no hard coding. hard coding in the robotics space looks like a roboticist sitting down, you know, describing to the robot how to move its arm, how to move its limb, what object to pick up where. Hard coding in the you know, robotics for the web space looks like a programmer sitting down, writing a program that says, you know, open up, open up this web page.53:58There is going to be a button that says search, you know, type this in that button, look at the third item, click this entry, then go here. this is where you need that millimeter accuracy that we were talking about on the maps. Like you need to like rule out everything and be very your language needs to be precise. Yeah, but the problem actually in both of those is that the world is too dynamic. if the world never changed, you could write this program once and your robot would do exactly what you programmed it to do.54:30And this is what we do on factory floors. on factory floors where you can control the environment, you can you can put in sensors, and what you need is repeatability. Sure, write all the programs that you want. But the world is extremely dynamic. Websites change, they So this is this is our is would you say it's like un un-tweb is unstructured in that way? Exactly. The web is highly unstructured, the web is highly diverse. the web is an open open ecosystem. No single company owns it. It is not the walled garden of the Apple Play Store where Apple and Apple provides you certain APIs and enforces certain standards.55:06The web has, for any concept you can think of, it has 20 competing standards and 21 different ways of doing it with those standards. and you know, over the last 30 years, we've built layers of technology that is meant for human consumption. So the browsers are trying to show you information. And what we do is is is solve this problem in the pure in the most general way possible. Just look at the screen. The reason why humans can do this is because they can perceive the pixels.55:37They this information was meant to be consumed by humans in a visual manner. So it consume this information in a visual manner and act like a human when it is necessary. Our models But even like, but but but but yeah, I would add the caveat that every time like you know Facebook would do a redesign of its homepage, it would like completely rewire people's brains and it would still take them time to readjust to it. Sure. Yeah. and machines can relearn and update their their beliefs because our agents actually go out into the world hundreds of thousands of times.56:13They are learning from interactions with live websites. and so a changing world is actually fantastic for our it actually reinforces our thesis. if the world is changing fast enough, you just have to keep getting data from that world and keep making updates to it. after a point, you know, even if the the layout of the website changes, like I use my agents for a lot of websites that are poorly designed, my web agents are superhuman compared to me.56:43It takes me a while to figure out where the button went, but they can spot things that I can't and they can figure things out. They're more persistent, they're a bit frustrated by bad design. And so we we send our computer use agents and our browser use agents to act on our behalf all the time. So I've had you for basically an hour at this point. So we'll we'll close on this and I'm gonna ask a an equally philosophical question as the one that I started with. Are you ready? Sure, let's do it. What is the capital of France? It's a good one.57:15The the philosophical response is that it's a good question that we should give further thought to. It's a bit of a compound. Well, this has been amazing. We'll have to have you back at some point. thank you so much for joining us. Thank you for having me, Brian. This was wonderful. it was offbeat and different, but but thoroughly enjoyable. Thanks again to Dreven Utori. If you're enjoying the show, you can help us out by liking and subscribing and checking out our newsletter.57:49You'll find that over at automated.fm. Thank you for tuning in. We will see you soon for another episode of Automated.
