This transcript is generated with the help of AI and is lightly edited for clarity.
//
CHELSEA:
I think the biggest risk is that everyone fails, that robotics as a whole fails, because robotics is so hard. There are so many pieces you have to put in place for anything to work.
When you’re watching a robot demo, if there are details about how it was done, it’s really important to read those and actually understand how that was developed. Was it developed in a way that is going to, in the long run, stand the test of time and be scalable?
Pi models are already running in production. That gives me optimism that we are at the point where this technology is mature enough to be useful.
//
REID:
Most of the AI revolution has happened behind glass—in search boxes, chat windows, image generators. The world we actually live in is made of objects that slip, doors that stick, and rooms no model has seen.
ARIA:
That’s why robotics remains one of the deepest tests of intelligence. It’s one thing to describe a warehouse; it’s another to walk into an unfamiliar building, read a new label, pack a box it’s never seen, and recover when something goes wrong—without being hand-programmed for any of it.
REID:
Chelsea Finn has worked on that problem from both sides of the frontier. At Stanford, her research on meta-learning helped define one of the central questions in modern AI. Her MAML paper has been cited tens of thousands of times.
ARIA:
At Physical Intelligence, the company she co-founded in 2024, that question has gotten very literal. Can a single model generalize broadly enough that robots don’t need to be reprogrammed for every task, every warehouse, every failure mode?
REID:
The factual version of that story includes months of 0% success rates on laundry folding before a single architectural insight unlocked the capability. Pi has since demonstrated a sequence of increasingly capable generalist policies.
ARIA:
Today’s conversation is about what it will take for AI to leave the screen—and what that transition reveals about intelligence itself.
REID:
Chelsea Finn, welcome to Possible. You’ve said that folding laundry is the most impressive thing you’ve seen a robot do. I want to start not with the result, but with the months of 0% success rates before it happened. What were those failures actually teaching you?
CHELSEA:
Yeah, so I think the first thing I’ll say is that robotics is really, really hard. And it’s really easy to underestimate how hard it is, because we are so good at manipulating all sorts of things around us with our hands—it comes as second nature. We don’t even think about how we go about flattening a shirt and folding it when we’re folding laundry. So it’s really easy to take for granted the fact that it’s not too hard for us to manipulate things. But actually, for a robot, you need to translate all of the sensor readings, all of the different RGB pixel values, into a vector of numbers—a large vector of numbers for all the different joints over time—for the robot to do.
And the thing that I think specifically I found about laundry is that there are so many different ways for even just a single shirt, or a single set of shirts, to be crumpled and configured. And dealing with that variability is very challenging, because the robot needs to understand how to translate all of these different configurations of a shirt into actions that will actually make progress on the task. And so one of the things we had found previously is that we were able to train robots to do tasks in narrow situations. And once it broadened out to be—even for a single shirt, but broadened to be a much wider range of configurations—the problem gets a lot harder.
And we started with something simple, and we had some results where if you start with the shirt flat, it’s able to fold it. And usually in research, it’s good to start with something that works, then make it incrementally harder. In this case, it was just a scenario where we went from a flat shirt to a crumpled shirt, made it way, way harder. And that’s where you do a little bit of banging your head against the wall for a few months before you actually start to see signs of life.
ARIA:
I mean, honestly, watching your videos, I was like, no—what if the shirt’s inside out?
CHELSEA:
They’re never going to be able to do it. And it’s like one of these things.
ARIA:
It’s so funny, because you’re like, wait, so a car can be self-driving and drive down the highway at 60 miles an hour, but the robot can’t fold the shirt? It’s just such an interesting disconnect. And so you’re taking on the physical world. And like you said, the physical world is so hard. What made you first think, okay, physical intelligence is a whole nother discipline—and that’s where you wanted to be, you wanted to be in the real world with physical intelligence? And maybe, what’s your definition of physical intelligence?
CHELSEA:
Yeah, so when I started working on robotics, it was maybe 11 years ago, 12 years ago at this point, and I was really fascinated by this problem of AI. How do we develop computers that are intelligent, that show the intelligence that people have? It’s just still, I think, so fascinating, and also has so much potential to have an impact on the world—and is having an impact on the world today. And I felt like, at least in terms of how the field was organized at the time, that different areas of AI were often solving a small subset of the problem. Like computer vision was really focused on object recognition, or object classification, or detecting text in the world, or various things like that.
I did a project on text detection, ultimately with the goal of helping visually impaired people, when I was an undergrad, for example, working on computer vision. And then even in natural language processing, there was semantic parsing and so forth. And I was frustrated, or dissatisfied, by the fact that these fields were organized around something that wasn’t the end problem. It was only a part of the problem. It was a means to an end, not the actual full thing. And I really wanted to work on something that was encapsulating the entirety of a problem, because I think that when you actually look at the entirety of the problem, you solve it in a different way than if you were to try to break it up into a subproblem.
And so that was one of the things that I found really appealing about robotics. I also find that the real world—there are so many challenges with it when you actually are faced with not just building a brain that processes images, but actually building a brain that uses that to translate it into actions and actually have a physical impact on the world. So those were some of the things that really drew me to robotics in particular. And then, in terms of how to actually think about this notion of physical intelligence, I think it’s really the ability to control any physically actuated device in any way that that device is physically capable of doing.
And I think that right now we have all these devices around us, ranging from dishwashers to laundry machines to Roombas to, eventually, robot arms, cars, and so forth. And we actually have so much ability to design all these different physical devices that are actuated, have motors and so forth—but the bottleneck is always actually making those smart, making them know how to accomplish some goal. And so I think that if we have physical intelligence, we’ll be able to breathe intelligence into all of those different devices, and basically any physical mechanism that we could imagine.
REID:
One of the things you’ve been precise about is that scale is necessary but not sufficient. Industrial data has no diversity. YouTube has an embodiment gap. Simulation is not real. And that’s a pretty thorough indictment of the obvious sources. So what does the right data strategy look like for this project?
CHELSEA:
So I think that in terms of machine learning, the first thing that you learn in machine learning classes is that you want your training data set to match the distribution of your test data set. And if the training data that you’re collecting is reflective of the scenarios that you’re going to see at test time, then machine learning will work. And if there’s a mismatch between those distributions, then all bets are off. And so the principle there is that we want the data that we’re training robots on to reflect the real-world situations that they’re going to be evaluated on. And I think that there is no getting around that. I think that other data sources can supplement that and help provide knowledge to robots.
But this kind of gets back to: robotics is hard, and you have to solve the hard problems to really make it work. And if we can scale large amounts of data of real robots in the real world doing real jobs, then that’s going to be the data that will fuel large foundation models of robots doing tasks effectively.
REID:
Well, and obviously having the shirts be crumpled versus straight is a very good micro example of that. So, language models have a natural training signal: predict the next token in a massive corpus. What’s the equivalent organizing principle for physical AI?
CHELSEA:
So I don’t think that there’s necessarily a direct analog per se between language model objectives and robot objectives. And people have tried to make analogs as well. You really want to be optimizing for what the robot is going to be tested for. And I think one of the convenient things about next-token prediction is that you can actually frame a lot of useful virtual assistant, chatbot, and translation tasks as next-token prediction tasks. And I think that we want to be in the world where we’re framing real robot tasks as an objective and a task for training these machine learning systems. And that likely means something that is going to be predicting actions and outputting actions, because at the end of the day, the robot needs to figure out how to control its motors to accomplish a task.
One other thing that I think is perhaps interesting, that has been a bit of a guiding principle in natural language, is that even data that is not really directly doing the task can be very useful. So next-token prediction on low-quality internet data can actually play a large role in pre-training large language models. And I think that we’ve actually started to see some of the same principle hold for robotics, where lower-quality robot data actually can also play a large role and actually improve performance of a downstream robot model—even compared to if you exclude the lower-quality data and only include the higher-quality data. And the interesting thing there, I think, is it might not work for the same reason that it works in language models.
But one of the things that we found—my intuition behind it is that if you show low-quality data of folding a shirt, for example, maybe the shirt was folded using a different strategy, maybe the final quality of the fold was not very good, maybe it was slower at completing the task. But when you add more diversity to data with different strategies, you’ll actually see more variations of how the task is completed. And then if the robot makes a mistake, it might actually end up in one of those variations that was only seen in the low-quality data. And that low-quality data shows it how to still make progress on the task. And so essentially it gives you more diversity of inputs to the model.
And the wider coverage you have, and the more diverse data you have, the better your model is going to be able to generalize and handle open-world conditions.
ARIA:
So you literally wrote one of the definitive papers on MAML. I think we said it was cited 40,000 times. And MAML was about adapting quickly to new tasks from prior experience. And Pi—Physical Intelligence, for our listeners—is doing something similar, but at a totally different scale. How much of the original thinking is still in what you’re building now, and what has changed?
CHELSEA:
Yeah, so I started working on meta-learning during my PhD, because when I was working on robotics, I was frustrated by the fact that we would train every task independently from scratch. So we would train the robot to hang up a shirt on a coat rack, and then we would just throw everything away and then train the robot to insert a cap onto a bottle, and then repeat that process. And there was nothing shared across the tasks. There was no reusability. And you would think that after you’ve learned some base set of motor skills, you should be able to learn the next task more quickly. And I think that certainly we’re still using the same sort of principle of how do we reuse across tasks.
And that’s really one of the central theses of the company: can we build a general-purpose model? And that building a general-purpose model will actually be easier than trying to tackle an individual, single, narrow task. And that’s also what has driven progress in language models. If you want to develop a really good machine translation system, you don’t only collect data from machine translation and train only on that. You start with a really powerful general language model. And actually, oftentimes now the general models are more effective at the specialized tasks than things that are special-purpose-built for that task.
And I think that the same principles will hold for robotics, for actually the reason that we talked about before, where it gives you more diverse data and more coverage over scenarios. So yeah, this sort of general pre-training I think is really important, and this reusability across tasks. Now, going back to meta-learning: the idea behind meta-learning isn’t just to be able to generalize to new tasks, it’s also to be able to adapt. I think we want to see that in robotics as well. And we’ve seen it to some extent with pre-training and fine-tuning. But I think that we also want robots to be able to adapt really efficiently to new environments, to new circumstances.
We’ve seen small proofs of concept of this in robotics, where in one of our recent projects we found that a robot was able to adapt to open a fridge where it was actually ambiguous whether the left side or the right side was the opening one. There wasn’t a handle. It tried to open the right side, it was failing, realized that it needed to try the left side instead, moved over to the left side, and successfully opened it from the left side. And so it’s able to do this sort of in-context adaptation, or really fast adaptation on the fly. We haven’t yet seen this proven out on a really large scale, where the robot can arbitrarily adapt to new circumstances and to new tasks.
But I think that we are well on our way towards that, and I’m optimistic about developing that sort of capability in robots in the future.
REID:
You know, I think we’ve all had that experience with a refrigerator of going, oh wait, it’s not this side, it’s the other side. So it’s actually a good parallel. So let’s dive in a little bit for our listeners who may not be as familiar with what the system for robots is. So, a robot walks into a warehouse or kitchen it’s never seen. What is actually adapting in that moment—the weights, the plan, the way it’s reading language? What’s the set of things in terms of the way it’s operating in a totally new environment?
CHELSEA:
Yeah, so the way that we approach the problem—like models such as ChatGPT, Gemini, and so forth—is to train a big neural network that takes as input a sequence of images and a language command, and potentially some other information like the joint readings, and then outputs how it wants the motors to move, and specifically what you want the angle of each joint to be for the next half second or so. And actually there are multiple trajectories that it predicts. And so a lot of the adaptation, or a lot of the generalization to different scenarios, is happening within this neural network. And so it’s implicit. It’s not explicitly broken down in any way. And then from there, in order to handle a new environment, there’s actually a lot that the model has to do.
It has to be able to first implicitly interpret where objects are, interpret the 3D position of those objects to some extent. The height of the table can vary across environments, the lighting conditions can vary, which affects perception. It needs to also be able to translate a language instruction of what to do into how that relates to the perception, and ultimately figure out how physically it should move its arm in a different way for this new environment. And that all happens just within the neural network. Now, one of the beautiful things about using end-to-end neural networks for this is that you don’t need to explicitly have a 3D model of the whole environment.
And I think that surely people don’t form a 3D mental model of their environment to complete a task. There are shortcuts you can take. And so that allows these systems to—it’s kind of the whole principle behind end-to-end training. It allows you to actually more effectively do the task, because you don’t have to accurately predict exactly what is the friction of the table, or what is the center of mass of a water bottle, and so forth.
ARIA:
So it’s funny, I’ve been recently making my kids make their own peanut butter and jelly sandwiches, and you would think that would be straightforward, but you’ve never watched an eight-year-old with a knife just being unable to spread peanut butter on a piece of bread. And so I imagine it’s sort of similar to when you’re watching the robots.
CHELSEA:
And you’re like, oh my God, do it. No, not that.
ARIA:
Like, oh, pick it up. And so, one of the videos, you had shown a robot its task. It had to put some dishes in a sink, and then it was asked to put a spatula away in the drawer. Sort of a simple task—even my kids could do that. And instead it opened the oven and put the spatula in the oven, because, oh, it’s a drawer, I’m going to open the oven and put it in. That feels like a bug. Is this a window into what the mental model of the robot is? Or what does that tell us about how these robots are seeing the world?
CHELSEA:
Yeah, I guess a few notes. So we actually hadn’t collected any data with ovens before, because we weren’t yet at the point where we were ready to start cooking things in the oven. And so whenever it saw a handle, it was overgeneralizing to different circumstances, where it assumed that if it saw a horizontal handle, that looks a lot like a drawer. And I think there’s actually—I’m not a neuroscience or psychology expert, but I think there are actually behavioral studies that show that at early ages, people also have a tendency to overgeneralize concepts in ways that aren’t correct.
And I think it’s actually in some ways a positive sign, because it means that the model has some notion of invariances, and you’d much rather that than getting the opposite of that, which is overfitting, where it can only open one drawer and it can’t handle any other drawer. So that’s a first note on that. The other thing that I’ll remark is we’ve also seen really interesting notes of generalization in other circumstances as well. So one recent example—we actually haven’t shown this publicly on any blog post or anything, but I found it really interesting—is that we were recently trying to get the robots to assemble a pinwheel, which involves a piece of paper that’s precut in a certain way.
There’s a pin, there’s a stick. You need to put the pin in the center of the paper, and then fold the four flaps, and then attach the stick to it. And we had collected pretty high-quality data, but with one particular strategy for doing the task, where it always involved picking up the paper with the left gripper and picking up the pin with the right, inserting the pin in that fashion. And one of the things that we found is that the robot actually made a mistake and it dropped the pin, and it dropped the pin on the other side of the table.
And we found that, in that circumstance, what the robot did is it actually picked up the pin with its left gripper and inserted it into the paper using its left gripper. And we actually never showed it any data of how to translate that sort of idea, that notion of completing the task with its left arm. So it seems like it actually learned this sort of equivariance between the right arm and the left arm. And we never explicitly told it that you can do everything with your right arm that you can with your left arm and so forth. It learned this sort of equivariance between them. And I think it’s another example of generalization—in this case, maybe it’s not overgeneralization, because it’s able to do it.
And it’s actually less than a one-millimeter hole to insert into. So pretty precise as well. And I think it’s a sign that these models are really learning about what are the things that are constant and invariant across different circumstances, and what are the things that vary.
REID:
So you went from a 20% language-following rate to 80% by changing how the action head attaches to the vision-language backbone. That’s a surprising result—the architecture affecting whether or not the robot understands what you’re saying. What does it tell you about how fragile language grounding is in physical systems?
CHELSEA:
So there are a lot of bits that go into getting a robot to follow instructions. And the biggest bit actually has to do with the data. So neural network systems are trying to find correlations in the data. And there are a lot of circumstances where you actually don’t need to look at the instruction in order to know what to do. You can just look at the scene, and it’s really obvious that you’re going to be asked to do something just based off of the scene. And because of that—and this is especially true, this is true in general—oftentimes if there’s a bunch of dirty dishes in front of you, then probably a good thing to do is to clean the dishes.
Or if there’s a bed that is unmade in front of you, probably you should make it. Or if there’s something that’s disassembled and needs to be assembled, probably you need to assemble it. So this is generally true. And it can be exacerbated by the way in which you collect the data, where, if you collect the data in a way where people always do the task in a certain way, or they set up the scene in a certain way and so forth, it can basically create these really strong correlations between the initial scene and what the robot does, in a way that the model will just learn to ignore the language instruction and just do the task based off of the image. This is the correct thing to do based on the training objective.
But it’s not exactly what we want, because we really want it to do exactly what we tell it to do. So there are a couple of things that we found to be helpful for this. One is to try to actually decorrelate the data and find ways where you have to look at the instruction in order to figure out what to do. And then the second, which you mentioned, is actually a change in the architecture and the way we train the models, which is that the vision-language models—which are basically like language models but with a vision encoder—they are actually really good at language following. And actually sometimes they have the opposite problem, where sometimes they ignore the image, because you can just answer a question without even looking at the image.
And so you could actually leverage the biases of these pre-trained vision-language models, which are really good at paying attention to language, when using them with robot data. And so the key idea there was to try to more natively plug into the vision-language model using tokenized actions rather than using continuous actions to train the backbone of the model. And we separately have a separate diffusion head, which is the way in which you often train image generative models, basically to predict continuous actions rather than these more coarse discretized actions. And we don’t use the gradients of that signal to train the backbone of the model.
REID:
You’ve argued that reinforcement learning is the physical AI equivalent of synthetic data in language models—robots learning from their own attempts rather than human demos. How far does that analogy go?
CHELSEA:
Yeah, so the most important bit, and the most useful bit, of this analogy is that synthetic data in language models is incredibly scalable. People are actually starting to complain that the internet isn’t big enough. And if the language model can generate its own data, then you’re basically just turning compute into data. And if you can find ways to scale up compute, then you can find ways to scale up the data source that you’re learning from and get more and more powerful models, and that can recursively feed into the model and the strength of the model in different ways.
And I think that analogously, when robots are attempting to do tasks, if they are doing the tasks autonomously and acting on their own accord and attempting the task themselves, learning from that data, that data is also going to be incredibly scalable—and more scalable than, for example, trying to teleoperate robots in a fully human-supervised fashion. So I think it’s a data source that, as these models start to develop a base level of proficiency, should be a really massively scalable data source for training these models. And that’s especially valuable in robotics, where you don’t have an internet of robot motor control data to start with. So I think that that’s the most important point. Now, there are challenges.
Even if the robot is acting autonomously, you need to make sure the hardware is reliable and doesn’t break down, or that there are safety precautions in place so that the robot doesn’t damage itself or damage its environment. And it does also still need to be interacting with the real world to be maximally useful. Whereas in some language model scenarios, you actually do a self-play thing where it talks to itself, for example, rather than talking to a person. And so there are some differences, but the most important bit is that it’s a really scalable data source for training models.
ARIA:
I love the idea that robots, they’re just like us. They see a task, they don’t read the instructions, they jump right in. Like, I’m putting together an IKEA table, I know what I’m doing here. They make the same mistakes.
REID:
RTFM.
ARIA:
And so, when you’re a person who’s not familiar with robotics, if they’re watching one of these Pi videos, what should they be asking to understand what it proves? What questions should they be asking if they’re watching a video to understand, oh, this is so exciting because it generalized the right-hand instructions to the left? How can we know how good we are from watching these videos?
CHELSEA:
It’s really hard to interpret robot demos. And part of that is also that it’s not too hard to fake a demo, and to show a robot doing something really impressive but not actually tell you how it was done. And if you aren’t told how it’s done, it may have been done in a way that actually isn’t, in the long term, going to work, or isn’t impressive. And so I think that when watching a robot demo, if there are details about how it was done, it’s really important to read those and actually understand how that was developed.
Was it developed in a way that is going to, in the long run, stand the test of time and be scalable and be something that could be repeated for many circumstances? Or was it something that was heavily engineered and required a lot of manpower to get that one task?
ARIA:
How novel was the task? Did the turkey have to be exactly there on the plate next to the mayo? It feels like you can fake it.
CHELSEA:
Yeah. And in these videos, you don’t see the answers to those questions. You don’t know, oh, if I touch this a little bit, what would happen? Or if the robot was in a different scene, and so forth. So it’s really hard as a starting point. And the text that goes along with these videos can provide a lot of context for why it’s interesting and why it’s impressive. One of the most challenging things that we have found to convey is actually generalization, because generalization is a property of the training data. And we can tell you that this is a home that the robot has literally never been in before, but the video doesn’t show that, because you don’t know what was in the data and what wasn’t.
And so we try to find ways to show that by actually showing us literally bringing the robot into the home and assembling it there and then running it, or showing videos of the robot not just working in one home but multiple homes, and doing many tasks in many rooms in many homes, to really try to illustrate that this was truly generalizing to those environments. Another example of this is, if you want to see reliability, do you just see one video? Or do you see a time lapse of the robot doing it continuously in an uncut shot?
We often have been trying to aim towards uncut videos as well, which really show over time what it’s doing. But it’s really hard. And to back up, in language models, I feel like you can really understand how well it works by actually interacting with it. With robots, we’re not there yet, because you need a robot in order to interact with it. But at the same time, I think that we have already started to see people play with these models. So we open-sourced the π₀.₅ model, and people are actually using it and building on it, which is really, really cool. And I think that it’s a testament to the fact that these models are actually useful and are actually improving on the state of the art. Insofar as you can actually interact with it—that’s when you get a real understanding of how well it works.
REID:
And part of the thing that’s very interesting about the generalization capabilities—building up baseline learnings, composability, et cetera—it’s not just technically interesting, but it’s also business-model interesting. So Pi’s argument is that every robotics application has historically required building a company around that one application, and a general-purpose model layer changes that. So what are the challenges to navigate that, and what’s the strongest argument against the general thesis?
CHELSEA:
The thesis feels so obvious to me right now. I think the hardest bit is that I do think that robotics is hard, and actually deploying robots into the real world is also really hard. And I think the biggest risk is that everyone fails, that robotics as a whole fails, and that even this approach won’t get us there, because robotics is so hard—there are so many pieces you have to put in place for anything to work. So that’s my first thought. A second thought is that for things like ChatGPT and software, you get distribution so easily just by putting things on the internet. And so many people have computers, so many people have phones and can interact directly with these systems.
And in robotics, we need distribution—you need to find ways to distribute. And I think that in some ways, maybe things like self-driving cars will be the closer analogy to the rollout of this technology. And it won’t be immediate. We might not have a ChatGPT moment where a huge population of people are interacting with these systems. So those are maybe my biggest points. I think that the problem is really hard. You need really reliable hardware. And even today we don’t have hardware that is as reliable as a car is, for example, and so forth. So there are so many aspects of it that are hard, let alone the completely unsolved problem of developing intelligence as well.
ARIA:
So we were just talking about how it’s hard to view a video of a robot and know, was it a novel situation? What was gamed? How did we do it? But you guys fine-tuned your model on a robot you’d never seen, from data you received remotely, without even knowing exactly how its actions were represented. And it worked. So what does that actually prove? Because you did a lot of that hard stuff.
CHELSEA:
Yeah. So one thing that we found that I think is perhaps surprising is that the ability to generalize to different robot platforms and robot embodiments is surprisingly easy. So this was actually technically not the first time that we had done this. We had previously, even before starting Physical Intelligence, a project where we were specifically trying to train models on multiple embodiments. In the past, people had only trained models for one robot platform, and assumed that if you’re trying to get it to work across lots of platforms, that would be really, really hard. And in that project—I mean, there are definitely challenges—but one of the things that we found was that it ended up being a lot easier than we expected.
And we took a model, we iterated on it a bit to train it on multiple platforms, and then we scaled up that model to a bigger model. And when we scaled it up, we actually didn’t tune the hyperparameters at all. We just changed the data mixture. We had previously just trained on one platform; we changed it to train on multiple platforms. We trained one model, sent that model to our collaborators at different universities. They ran it on their robots, and in most of the scenarios, the model that we sent them was better than the model that they had developed for their project on their robot.
ARIA:
And so just so people understand, this is different universities who have different robots that do not look the same, have different specifications, and they’re using your technology and it is still performing better than what they had trained in-house.
CHELSEA:
Exactly. And the robots, they look different. They might have different numbers of joints, they might be larger or smaller. Their cameras are set up in different ways too—where they’re mounted, even completely different orientations of the cameras, the height of the table, the setup. There are so many different things. And one of the things that we found is, yeah, if you have an expressive enough neural network model that can fit lots of data, and you feed in data from lots of embodiments, the model is really good at just being able to handle this. And in some ways, people are really good at this too.
We don’t just control our own body, but we can control cars, we can control video game characters, and we can very quickly, with a bit of data, learn how to control all of those different things. And it’s analogous to that.
REID:
Although it is interesting to watch people who are not familiar with video games try to do video game characters. But yes. So Pi’s been clear that there’s no simple commercialization timeline. So what do the challenges, risks, and possible failure look like? These long-horizon bets can sometimes be very difficult to pull off.
CHELSEA:
Yeah. So certainly long-horizon bets are very difficult to pull off—all sorts of challenges as well. The one thing I will say that makes me optimistic—I’ve been talking a lot about how robots are hard—but I will say that Pi models are already running in production on multiple different platforms, for one of our own robots but also for other partner companies as well. And so that, I think, gives me optimism that we are at the point where this technology is mature enough to be useful. There’s still a long way to go, but I think that definitely gives some optimism that this technology is starting to work. In terms of the challenges and the different routes—
Like I’ve mentioned before, we think that solving this intelligence problem that hasn’t been solved before is the biggest challenge. And that is the thing that we need to focus on. And so we’re orienting ourselves really around that problem and doing things that will help us advance the intelligence of the models—whether that be, of course, doing a lot of research and collecting data and so forth and studying those models, making them stronger and stronger. But also sometimes we find that hardware is a bottleneck, and we need to actually improve the reliability of the hardware we’re working on, or the teleop device or something like that, because that’s bottlenecking our ability to make progress as quickly as possible. So we’re really oriented around that.
We feel like if we take on customers now, or too early, that will slow us down and make it harder to address that core scientific and technological question. At the same time, I think that one thing that’s really nice about this is that the model will get stronger if it has real data, and data of real use cases. And so it’s actually very well aligned to continue to develop the intelligence of these models and also figure out how to deploy robots. Because if we’re deploying robots and getting real data, that’s going to make the model stronger. And so I think it’s actually—while we’re not focused on revenue, getting customers and so forth, we are trying to deploy robots and using that to get good data.
And so we’re both learning about how to actually make this stuff useful and how to make the model stronger at the same time.
ARIA:
So, Professor Maja Matarić, who we had on Possible, was talking a lot about robots in the physical form. And what she talked about, which sort of seems obvious, is that the more humanoid a robot is, the more human expectations it carries. And especially if the robot is big—people respond differently if a robot is your size, if it’s three-quarters of your size, if it’s small. All of that carries sort of expectations, and also different levels of scariness. We’ve seen the Jetsons’ housekeeper, who is humanoid. We’ve seen the Terminator, who is humanoid. No one’s really upset about a Roomba. There are different expectations that come with different kinds of robots. And so you’ve worked across all sorts of different embodiments of robots.
How do you think about the humanoid bet? Is it definitely going to be humanoids? Is it definitely not? Somewhere in between? What is your thinking?
CHELSEA:
Humanoids are a lot of fun. I actually have one in my office at Stanford. We’ve done some projects with it at Stanford as well. I think that it’s really cool. They’re just cool. So there’s that. And then I think it’ll be one of the embodiments that are deployed. I think that there are a lot of things that are nice about them. At the same time, I think that there are a lot of things that are really unappealing about them as well. So I talked about how robots are not very reliable, like the hardware, and in particular, the price of robots has been dropping substantially.
And you can buy robot arms that are far, far cheaper—orders of magnitude cheaper than they were 10 years ago—which is amazing. But these robots, while they are fairly inexpensive, they break all the time. And this is even for just a single arm with six motors, a gripper, and an RGB camera. And as you make a hardware system more complex, it becomes less and less reliable. And so if you’re going from—even if six motors and two cameras are unreliable, then something like a humanoid is off the charts unreliable. And even for the humanoid that we have at Stanford, it actually takes two to three people to do a single experiment on it.
In order to safely do an experiment, ensure that it’s safe, ensure that it’s not going to overheat—really easy to overheat motors as well. And so if we really want to solve this intelligence problem, we think that we can move fastest by starting with simple systems, because simple systems are already incredibly capable. You can do so many things. We’ve actually been surprised by some of the things that we can do with just single-motor grippers and so forth. We trained a robot to light a match, for example, without using any force feedback or tactile feedback. We’ve been able to teleoperate robots to empty dishwashers, to fold laundry. And it’s actually very hard to find tasks that you’re not able to do with these kinds of systems.
And so if you get the reliability benefit and the simplicity of the overall system, that’s a huge win. And if you’re still able to do all those capabilities, then it means that you’re able to move a lot faster on solving the intelligence part.
ARIA:
You said the price has dropped significantly in the last 10 years. That’s what we expect from technology. How has the reliability changed over the last 10 years? Stayed the same, gotten worse? Is that something else that might get better and better over time?
CHELSEA:
I think it should get a lot better. So there are more expensive robots that are more reliable. You can buy a $20,000 robot that is developed for industrial use cases and is quite reliable. But those robots, they’re not really designed for more dexterous, fine manipulation. They’re really designed for just repeating the same motion blindly over and over again in a factory. And so oftentimes, those design considerations—we still actually use some of those robots, because they’re nice for some things and they are reliable, but they’re not really designed in a way that we would want to use for a really broad range of tasks, because the intelligence hadn’t been developed yet. And so they were developed with the software technology in mind.
And so I think that I’m hopeful that the reliability should go up significantly. I’m not a hardware expert, but I don’t think there’s anything fundamentally new that has to be discovered or developed to make them more reliable. That doesn’t mean it’s not hard. I think it is really hard, but it just takes work and it’ll take time.
REID:
Well, comparing software and hardware—software AI gave us a culture of ship fast, measure, patch, based on the assumption that a bad output is annoying but not dangerous. There are obviously some asterisks around that, around if you’re talking to a young person about suicide or other kinds of things. But generally speaking, that is a kind of a culture of development. How do you think about the line between good enough to deploy and safe enough to deploy in the physical world, when those aren’t necessarily the same thing?
CHELSEA:
The main difference is physical safety, of course. And there are many principles in engineering that are really useful here, where if you have safety precautions that are redundant with each other, then you can put together a safe robot system even if some bits of the software—like the neural network that’s controlling it—are unpredictable or unreliable. And so once we have those systems in place, it allows you to develop these systems even if the neural network part is unreliable. It allows you to develop things that are safe and so forth. And we’ve taken safety seriously in a number of different circumstances. We’ve actually started to give our robots knives, for example, to slice zucchini. There are a lot of things that knives are very useful for, but we found ways to ensure that the people around the robot are safe and so forth. The last point that I wanted to mention is that one of the things that we found somewhat convenient is that if you use robots that are weak, that don’t have super strong motors, that also is another kind of built-in safety layer, because they’re just actually physically not capable of causing a lot of damage as well.
And so that’s one additional thing. And there are a lot of tasks that you can do without super strong motors, for example. I think that once you make a really strong robot, that no longer is the case, and you need to think even harder about some of these challenges. And some of the things we put in place for weak robots would also apply to strong robots. But that’s one additional redundancy aspect that we see in our robots.
ARIA:
I can’t wait for the social media clip that’s like, we’ve started giving our robots knives. Everyone’s going to freak out. There is obviously sort of a conversation, perhaps a backlash, about AI when we’re talking about LLMs. The conversation has been about knowledge workers, lawyers, coders, and doctors. And a lot of the backlash and conversation has been, why are we taking our jobs away? Why are we taking human agency away? What are people going to do in the future? And this hits a specific group of workers that wasn’t hit in the past by automation, wasn’t hit by machines and factories. And so when we’re talking about robots, this hits a different class of people. These are people who drive cars or clean houses.
And is there a different moral question that goes on if we’re going to be displacing this other group of workers, who are perhaps even further from the creation of this AI?
CHELSEA:
Yeah, lots of thoughts. First, I do think that language models are impacting the work of workers that are more blue-collar workers, for example—maybe not blue-collar is the right word, but people who are doing manual data entry, for example. I think that there are a lot of examples of things that are more digital but also don’t involve really high degrees of intelligence that I expect to be affected by language models, if they aren’t already. So I think it’s true for both technologies. I think that ideally we should find ways for these people to be part of the conversation. Another thing that makes me quite optimistic is that a lot of technology is not replacing people, but actually making people more productive and augmenting them.
And I think that we might see the same with robotics as well. And I think we’ve seen this with language models too. Even with the workforce we have, there are massive labor shortages. There are people doing jobs that are really unpleasant that they don’t want to be doing. There’s really high turnover for a lot of these jobs as well. And so I think that there’s a lot of potential to increase productivity overall, to increase quality of life. And then I also think that with language models—and I think we’ll see this with robots—people are actually using these models for things that they wouldn’t have done otherwise, that aren’t currently being done by a person.
And so I think that we’ll see that with robotics as well.
ARIA:
I think one of the things that I think about is that so much of the language model side is, we hope, some of it’s taking away drudgery. Like talking about manual data entry—it’s like, oh, I’m so excited I don’t have to do the manual data entry, so for my job I can focus on the stuff that’s important. And I just wish I knew the exact numbers. They were saying that the invention of the dishwasher and the vacuum freed women, mostly, from like 10 to 15 hours a week of manual labor that they were doing inside the home, which enabled them to work outside the home, hang out with their kids, have fun. And so you could imagine with robots, there is so much of that drudgery.
Even if it’s not dangerous, there’s just drudgery that could be taken care of. I mean, I am thankful every day for my Roomba, which means I have to vacuum my home less. And so I think there are better examples out there than just housework. But even that one—if you could save whatever, two, three hours a week from doing something, it would be a huge unlock for certain segments of people.
CHELSEA:
Yeah. I also think that you could imagine robots playing games with them, for example, that you wouldn’t otherwise be playing, or other forms of recreation.
REID:
What are some instances? I always think about how AI can be amplification intelligence and work with human beings. What are some examples that come top of mind in physical intelligence?
CHELSEA:
Yeah, I’m really excited about human-robot collaboration, and I think that that’s the most natural example of this. I’ve been trying to get some of my PhD students excited about it too. I think that any scenario where people are collaborating with each other is a very natural example. And one thing that comes to mind is, if you’re cooking, maybe in your own kitchen, can a robot retrieve ingredients? Can it chop ingredients to your specification? Can it clean up the dishes while you’re cooking, for example? And so that’s one example.
I also think that there’s actually even more direct collaboration you could imagine, where, for example, surgeons often are using robots, and if you could actually not have them fully control it, but actually help stabilize their motions, or help augment them to do it slightly more precisely, give them a lever to more precisely, more accurately, or maybe a little bit more quickly do a surgical task—then that’s something where both the AI and the human are physically controlling the robot.
REID:
One of the questions in all this stuff, which I believe is completely unanswered—although that’s why I’m asking the question—is who should be liable when a general-purpose robot creates harm in a context that its deployer didn’t anticipate? I’m not sure the legal frameworks that we have are even asking the right questions, because it’s obviously deployer, constructor, environment, et cetera. Any of this stuff that you’ve seen some early possible suggestions on?
CHELSEA:
I think that we have this notion of insurance companies, for example, that I think might translate to AI systems at some point, but that wouldn’t necessarily tell us the blame or liability. I also think that while we’re starting to see these models work in production, we still don’t know exactly the shape of the technology and the ways in which it’s going to be deployed. Is there going to be a base model and a deployer and so forth? Is it going to be more vertically integrated? I think we don’t know those yet. And I think we might be able to draw from other frameworks. For example, people build furniture that, if it’s in a way that could fall over, then there’s some liability there.
And so maybe there are some things that we might draw on there, or cars or autonomous vehicles and so forth. But yeah, definitely, I think it’s early days to figure it out.
ARIA:
So, sort of similar to that again—you guys are sort of pre-commercialization, your robots aren’t being used in the real world yet. But people always talk about surveillance, wars, border enforcement. And some of this is great—people talk about cameras being able to catch speeders, people talk about, can we make war safer for people. Do you think about the lines where you’re like, I’m drawing that line, I won’t go into defense? Or how do you think about that sticky moral question?
CHELSEA:
So maybe first—we actually have robots in production that are running our models, some of our robots and some robots from other companies as well. So it is actually there. We aren’t currently deploying in any of those applications or really pursuing any of them right now. I think that there are so many other applications that are really exciting and really impactful, and we haven’t yet been faced with a situation where we would really want to strongly consider any of them.
ARIA:
So we’ve been talking a lot about robots in the home. They can help you cook, they can clean. How many different robots do you think a person might have in the future? And this could be both in their personal life in the home, and also in jobs—various jobs might have many different types of robots that are helping them.
CHELSEA:
It’s really hard to predict, of course, but I think it will be more than one. I think that there will be a diversity of platforms. I think that there is something to economies of scale and the ability to really mass-produce devices. At the same time, we already see scenarios where we have a microwave for one thing and a dishwasher for another thing and so forth. And it seems like, as a global community, we have a way to manufacture all sorts of different devices and so forth. So I could imagine it being really diverse, where you have some that are really small and some that are large, and some that take a longer form factor versus a more compact form factor.
So yeah, I can definitely imagine a world where it’s very diverse. Maybe you have different robots that are in the home versus in workplace environments, grocery stores, and so forth.
REID:
So one of the things that I’ve thought a lot about on the chatbot side is how you might have a chatbot that goes with you your entire life. It’s like your companion, and it might be something that, for example, parents might even go, oh, we have a kid, and here’s the chatbot for the kid as a way of guiding and helping. Do you see something like that for robots? I mean, you’re doing this kind of general purpose—and the question of what could be a, you know, not invisible friend, a physical friend. And not just the one that might go through a person’s life with them, but also how robots will then play different roles as you get to different stages in your life.
CHELSEA:
Yeah, I think personalization first is super valuable, both in terms of efficiency and in terms of user satisfaction. You kind of expect it to know stuff about you and to use that to answer appropriately. And different people have wildly different preferences, including very different preferences around how you want things done physically as well. And I imagine that in the future, our robots should be capable of being at the very least prompted in the same way that we prompt a language model, where you give a very detailed prompt like, okay, I really like cooking, so please don’t cook any meals for me. But I dislike cleaning a lot, so please do the dishes whenever you see them not done.
And I don’t know, once a week, please make sure the trash goes out, and all these other things. So I think that that sort of customizability is something that I expect to see in these models, at the very least through prompting, if not through longer-term memory as well. And then I think that definitely I could imagine—I mean, within the realm of personal robots, there’s also the huge other realm of commercial robots and so forth. But for more personal robots, I think that there are extremely different tasks at different ages, ranging from changing diapers to helping someone get up off of the couch or out of bed. I think that there’s also really tremendous value that could be had, especially on the tail ends of those spectrums as well.
And maybe that’s also a good example of something where people might do something that they wouldn’t be doing otherwise. I think that if someone had a robot that could help them stay independent for longer in their own house, they probably would ask it to do stuff that they wouldn’t ordinarily ask a human caretaker to do, because they have privacy, because they don’t feel bad about imposing on that person and so forth—especially if that robot is stronger and doesn’t get fatigued, or is able to work at all hours of the day and so forth. So yeah, that’s, I think, one example application that is really cool and would look very different from a human doing that job.
REID:
So, rapid-fire. Is there a movie, song, or book that fills you with optimism for the future?
CHELSEA:
So, perhaps surprisingly, I hardly watch any movies, and I hardly ever read books. But I have a song, which is also probably very atypical, which is *Aurora Awakes*, composed by John Mackey. No lyrics. It’s a song that I have played as part of a wind ensemble in the past, and I find it to be—
REID:
What do you play?
CHELSEA:
I played trumpet for maybe almost 10 years, something like that.
REID:
So cool.
ARIA:
What is a question that you wish people would ask you more often?
CHELSEA:
I think anything technical. I love chatting about robots. Okay, maybe not all the time, but I think that that’s the most fun part of my job. I don’t like email as much, like probably most people. And if I get to actually be engaged with the technology and how to actually get something to work—whether it’s in the weeds and the details of where you’re positioning a camera, or how much data you’re training on and so forth—I really enjoy trying to get things to work well.
REID:
So where do you see progress or momentum outside your field that inspires you?
CHELSEA:
I think this is a hard one. I feel like I’m not an expert in other technical fields to really judge if the progress was truly impressive or not. I think things like the COVID vaccine development were really fantastic to see—seeing a large group of people really come together to develop something that could have such a massive impact. I think also outside of technical fields, I often find maybe not even fields, but individuals, to be quite inspiring, whether it be teachers who are really passionate about their job and who taught me a lot or were inspiring to me in various ways, or, I don’t know, competitive mountain climbers—
—or rock climbers that scale buildings and rocks that I would have no ability to do myself.
ARIA:
So our final question is, can you leave us with a final thought on what is possible to achieve in the next 15 years if everything breaks humanity’s way, and what’s the first step in that direction?
CHELSEA:
Naturally, I think toward robotics. If robotics is successful, I think that there is so much drudgery that could be taken off people—people will basically be freed of having to do things that they don’t want to do but have to get done. I think that people will be more productive. And if you’re doing stuff that you enjoy, you’re usually better at it as well. And so, the ability to have robots complete all sorts of tasks—and I think maybe even more so than helping people in first-world countries and homes, there are all sorts of other forms of labor exploitation and so forth that are also incredibly terrible to hear about. And I think there are so many opportunities for physical tasks to be completed by technology. And yeah, it’d be amazing to see that.
ARIA:
Awesome.
REID:
Thank you for joining us.
CHELSEA:
Yeah, thank you for having me.
ARIA:
Thanks.
REID:
Possible is produced by Palette Media. It’s hosted by Aria Finger and me, Reid Hoffman. Our showrunner is Shaun Young. Possible is produced by Thanasi Dilos, Katie Sanders, Spencer Strasmore, Yimu Xiu, Aman Suri, Danny Garrison, Trent Barboza, and Tafadzwa Nemarundwe.
ARIA:
Special thanks to Surya Yalamanchili, Saida Sapieva, Ian Alas, Greg Beato, Parth Patil, and Ben Relles.
And a big thanks to Rachel Rosenzweig, Camille Ricketts, Lachy Groom, and the team at Greylock.