This transcript is generated with the help of AI and is lightly edited for clarity.
//
KANJUN QIU:
What would the Ten Commandments for technologists look like?
I think it’s possible to have a digital world that we feel like we can change it, like anything in our house.
The only part of it I know what to do about is that I think I want to see if we can make software go from a renter’s economy to an owner’s economy.
//
REID:
While studying computer science at MIT, Kanjun Qiu paid her way through school by writing high-frequency trading algorithms to trade the stock market. Fast forward to today, and she’s raised $200 million from NVIDIA and others to build AI systems that can reason and code, with access to a GPU cluster that most companies can only dream about. But here’s what’s fascinating. She’s not racing to ship a product.
ARIA:
Kanjun is building something deeper, what she calls non-leaky abstractions. They make AI systems reliable enough to actually delegate work to. She co-founded The Archive, a co-living house for founders and AI researchers, where the conversations weren’t about the next funding round, but about the fundamentals of human intelligence that machines still lack. And that philosophy shapes the outlook at Imbue.
REID:
We’re going into territory that still too many AI conversations skip. How do we ensure humans and their creative work retain philosophical agency in the AI future, not just economic power? If we want users to truly own their AI tools, what does that require architecturally? And fundamentally, if everyone had a powerful personal AI agent, how would that change social dynamics and human development?
ARIA:
Without further ado, welcome to Possible, Kanjun Qiu. Kanjun, I am so delighted to be here with you today. And I would say that many of my smartest and most thoughtful friends are very big into meditation, and I would count you among that group. And so, as a meditation novice — or I don’t even know if you can say novice if you’ve never done it — as a meditation ignorant, you just came back from a meditation retreat. Tell us about it. How did it change your thinking? How was it?
KANJUN QIU:
Yeah, I did a really wonderful retreat. I’ve been kind of like poorly meditating for quite a while because being a founder is stressful. And what I learned is that the different styles have a lot of variance in how effective they are for me, and I think personally for other people. And so I just came back from a Mahamudra retreat, and I would say meditation is probably like one of the most effective ways to see reality clearly and like think clearly because, for me, what it seems to do — my model of it is that, you know, when we have our amygdala hijacked, we’re triggered with like a threat response. This happens a lot when something bad happens and it’s really scary and like things are stressful.
REID:
Which is more or less like every day in startups.
KANJUN QIU:
It’s like basically all the time. When that’s the case, it’s hard for me — and I think humans generally — to see outside the triggered event and so to kind of put the triggered event in context and not have it just take over our minds. And meditation, and this whole idea of resting in awareness and being the field of awareness — kind of what they call it — lets me see the thing that got triggered as like a relatively small thing in my field of awareness. Like, okay, that happened. Cool. There’s all this other information also to consider. And so now my brain can work as an information processing system and weigh information properly, as opposed to weighing all of the fear responses super, super strongly, which is like evolutionarily adaptive, but not that helpful.
ARIA:
And do you see — you know, you were, you know, you’re observing your own thoughts, you’re paying attention, you’re sort of moving things to the side that would have triggered an outsized response — does it make you think about the current state of building AI and how we think about how AI is thinking and reasoning and all of the things we’re trying to get models to do today?
KANJUN QIU:
AIs are a really strange intelligence. My model of how meditation works is that it lets us see the process that is kind of observing everything. So like normally, when we’re engaged in our day-to-day lives, we’re just kind of experiencing things. But it’s rare for us to be able to notice and look at the consciousness process that’s like noticing all of the sensory experience and like deciding to do stuff with it and like showing thoughts and like kind of seeing that. You know, in meditation they call it awareness. Awareness I think of as like a substrate that is consciousness — like consciousness substrate is composed of awareness — and like it is the thing that generates our perceptual field and our thoughts and things like that.
KANJUN QIU:
And like seeing that as the generator suddenly frees us from being kind of pulled into all of these thoughts and like, sense of self. And so I think AI systems don’t have such a thing. Like they don’t have this like fear trigger response that like suddenly changes the way that they attend to their kind of inputs. And so it makes me wonder, like, I guess like reinforcement learning from feedback is a form of meditation. It’s like a — yeah — borked form of meditation that gets something. Maybe it’s more — maybe reinforcement learning from human feedback is more like, kind of Instagram Reels
REID:
Or Pavlov.
KANJUN QIU:
Or Pavlov. That’s right.
REID:
We lack a bell here to ring. And one of the things that was really awesome as we just were kicking off this podcast is realizing that both you and I have done retreats with the same teacher, who is really good. And I was actually thinking about the same question myself because, you know, you have this question of, okay, so you have an aspiration, but you need to form it into an intention. And then you’re kind of reclaiming the conscious agency in terms of what you’re doing. And part of the meditative practice is to try to be self-possessed of your own mind in doing that and not rely on reflexes — amygdala is one of them — also preconceptions and everything else.
REID:
So you have a kind of calm focus on the thing you think you should be focused on. Now like part of the thing that I think it indicates is we’re at best only partial in the way that the training of these systems works: how are they, as it were, attending to their own focus as they’re doing their own training. Right. So we are providing that focus. It’s one of the reasons why I think most thoughtful people say, no, it’s not conscious yet, or it doesn’t have awareness yet — because it’s not providing that kind of feedback.
KANJUN QIU:
That’s very interesting. It’s not attending to its own attention mechanism.
REID:
Exactly. What would it take to be thinking that that was — that is what the AI is doing. Now, maybe it’s because you could see something in introspection currently in all these AI systems. The introspectability of these systems rounds to nothing. Right. But like what would that be? And then where would you see it and then how would you reflect it? And it’s not as simple as a Turing test. As an incident. Turing test got us a long ways along this journey. But it’s like, it’s much more. And that kind of awareness of it was one of the things that I thought I was also perhaps, pun intended, meditating upon.
KANJUN QIU:
That’s interesting. Like, in theory you could — I mean this probably wouldn’t work in training — but like you could pass it its own attention matrix and then see like, maybe it can notice that as part or like it can have that as part of its input stream.
REID:
Yeah. Where is the self that is observing?
KANJUN QIU:
Where is the self that is observing?
ARIA:
Shouldn’t AI go to therapy and learn that it’s too much of a people-pleaser and learn that it needs to like say no or stop saying I’m sorry?
REID:
It’s like in many of these things, it’s balance. You want it to be something of a people-pleaser.
ARIA:
Right. Of course, of course, of course.
REID:
Because if not, you know.
ARIA:
It’s just super annoying.
REID:
Most Hollywood films follow this path.
ARIA:
That’s what people are scared of. That’s what people are scared of.
REID:
Yeah. Like people-pleaser is, generally speaking, a good thing. It’s just the right kind.
ARIA:
Yeah, you gotta tune it.
REID:
To your better health.
ARIA:
So you are building Imbue and your focus is on building infrastructure for reliable AI agents. Tell us what Imbue is and what you hope to do.
KANJUN QIU:
Yeah, we started Imbue in 2021 and the purpose was at that time — it was kind of early, pre-ChatGPT — and we really felt like, okay, this generation of models is probably going to make it all the way, not like the past decade. And in that period of time, the primary discussion around AI was — and the concerns about AI were — focused on technical safety. And we really felt like there’s all this conversation about AI and technical safety and the kind of world you want to build, but like we want to build a world with AI that we want to live in. And magical, hand-wavy abundant future is like not very concrete.
KANJUN QIU:
And so we really wanted to think about what are the core counterfactual turning points where it makes a difference between the default future and a future that we could create, such that if we did those things then like we’ll end up in a better future. We really landed on the idea that for us it was about the decentralization of power and kind of how AI affects people’s — like the distribution of power — and also people’s sense of agency and authorship over their lives.
KANJUN QIU:
We think there’s a real potential that AI would result in a centralization of power, both economically and politically, and that there are some things that we can do right now, in the next like four years, to shift that dynamic so that the default results in a much more democratic kind of decentralized future where each person’s values, judgments, inputs, and decisions for their lives can be acted upon — that they can act upon it — as opposed to us living in a more panopticon system that maybe is the default path.
ARIA:
You started in 2021, which feels like a world away from now. Like you said, ChatGPT hadn’t been released. You thought we weren’t in an AI winter, but no one was really talking about AI from a mainstream perspective. What do you think in 2026? Like, what have you learned over the last five years? Like, were there inflection points that happened as you thought they would? Like, what is the difference between your thesis now and your thesis in 2021 when you launched?
KANJUN QIU:
To be honest, I think a lot of the things that we predicted would happen have roughly happened, roughly on the timescale that we expected. So we expected coding to become super — like the first thing to be automated — and to be like a — it’s a fully enclosed system that is testable and so perfect for training. When we rolled that out a little bit, we started to imagine, okay, everyone will be able to make software. People will be able to make more complex software. What is that going to look like for the world? The second thing we expected is that AI agents, roughly on this timeline, would start to become a thing.
KANJUN QIU:
Like these systems will slowly be able to do longer and longer horizon tasks and also they’re going to want your data to be able to do those tasks. And so we saw these kind of like bifurcations in the road. It took maybe a year or two to like really land on a thesis of software and a thesis of agents. The bifurcation for software I would think of as, right now we live in a software environment, a digital environment that’s actually a little bit alarming. Like, I put my phone in a different room at night because I might accidentally end up on it until like 2 a.m.
ARIA:
Sure. I don’t know what you’re talking about. Never happened to me. (laughs)
KANJUN QIU:
Never happened to you. I know. It’s really a me problem.
REID:
It’s an every night problem for all of us.
KANJUN QIU:
It’s an every night problem. I’m kind of like, you know, if there was literally any other piece of furniture I felt this way about, I would replace that piece of furniture. Like my sofa, my bed. But I can’t replace my phone. And so why do we feel this way about our phones? It’s because our phones have these objects on them — all these apps built by people whose incentives are not fully aligned with ours. They’re not misaligned, but they’re not fully aligned. And so it’s actually a really good example of what happens when things we depend on, that we are really intimate with, that can immediately get our attention throughout the day — every day — we’re really intimate with these things.
KANJUN QIU:
What happens when these things we depend on are built by people whose incentives are not fully aligned with ours? There’s this divergence over time toward value capture. And with agents, that divergence can be more extreme. It could be possible to have a world where there are just a few agents — like a Facebook agent, an OpenAI agent, an Anthropic agent, and maybe 10 more. And so that’s not that many. Maybe just like with social media systems, we put all of our data into one or two of these agents. It’s hard to get that data out. The agent gets smarter and smarter about each of us because it understands more about us, it’s seen more of our data.
KANJUN QIU:
And because we put in that data one at a time, there’s this like, slippery slope toward all of a sudden these couple of agents have all of my data and have seen everything about me. And so if I switch to a different agent, it’s just not going to work that well. So I’ll just stick with these. And so then you have this lock-in. And then I, as a person, am suddenly stuck. It’s not a competitive market anymore. And if I’m stuck then it’s really easy for me to be kind of extracted from — again, not like in an evil way — but extracted from because these are profit-seeking companies that operate in an ROI environment. And so that’s kind of a future I don’t want to end up in.
REID:
When we go through life, we’re never fully aligned with anything. We’re never fully aligned. Like Aria and I work together a whole bunch. We’re not fully aligned. You and I have done a number of these things together. We’re not fully aligned. So full alignment I don’t see anywhere in my life. I don’t see it with my black Labrador. I don’t see it — like it’s like this whole thing. Where do you see these alignments and what are the things that you are doing with Imbue to try to have this sufficiency of good alignments and also avoid bad misalignments?
KANJUN QIU:
That’s a great question. So I actually agree with you that like nothing is fully aligned. However, the more intimate something is to us, the more aligned we want it to be. Like we want to be super aligned with our partner. Otherwise there are problems that arise over time because we kind of build our lives around it. Similarly, I personally think we want to be super aligned with our sources of information and like our information systems because the information we receive as a person like shapes our understanding of reality and our model of the world, and that shapes our action. Like my model of the world and how it works is going to completely affect and change what actions I take in life.
KANJUN QIU:
And so there’s a way in which if the information sources I’m getting are not quite aligned with me and they’re maybe not quite truthful and they’re not quite telling, then I might end up making really different life choices subtly. And I think people are doing that because of Twitter and social media algorithms and things like that. And so the more intimate something is, the more, kind of, we interface with it and like take its information or effect seriously, the more we want it to be aligned. And so your question is like, okay, how do we — like what’s necessary — and how do we end up in a world where it like actually feels really good?
KANJUN QIU:
And I think it’s likely the case that these systems are never like fully aligned with us, but that what we want is — at least in my mind, what I want — is for people to have more choice and authorship and more accurate models of reality so that they can make better decisions about their lives. And so what that means is I think we want systems — at least for me, what I want — is systems that are open enough that I can inspect them or that people can inspect them, and so I can develop deep trust in them. I have a preference for systems that operate locally on local data and don’t send that to a central server. And maybe it’s okay for them to send to a central server, that’s okay. But like, I want to have my data.
KANJUN QIU:
Like, I don’t love the fact that ChatGPT stores memories about me that I can’t export. I want to have my data so that I can move to a different system. And this is in the kind of policy landscape called data interoperability. And I think what that allows — the ability to move our data around — is it allows for competition, and competition allows for markets that end up being more aligned with us, like products that are more aligned with us, the user. And so what we want, I think, is like a lot of competition and a really good ability to move between different agents and like have open versions, have versions that can be like inspected fully and explained, etc. And that will let us at least get to more aligned versions of these systems and that will be better.
ARIA:
It’s funny, in the most simplistic terms, this reminds me — and I candidly, I don’t even know what year it was, and I don’t even know who did it — but you used to not be able to change your cell phone number. If you went from Verizon to AT&T, you lost your number. And that was like a huge, huge issue for switching costs, and people were locked in. And it feels like, again, that’s just like the most basic form of like, can you take your data with you? And it was actually really good when the government said like, no, you can take your phone number when you go to a different service. So that’s super interesting, and I feel like a lot of what you’re saying — and please correct me if this is not what you’re saying —
ARIA:
People talk about LLMs right now as a black box. It’s like, how can I trust it if I don’t know how it got here? I can’t inspect it. I can’t see the lines of code that got us here. And so there’s no way to build that trust if I don’t know what’s going on. And so it seems like what you want to do is let’s build these agents that you can trust them, they can reason, they can code. They are — you know, they’re reliable, they’re going to do what you think they’re going to do at the right time. And so that’s why they can become, you know, the software engineer of the future.
ARIA:
And so is that what you think is the best-case scenario, getting rid of the black box, having them be reliable, and then we can truly have them as coworkers, as agents and things to work with, you know, on a daily basis?
KANJUN QIU:
Yeah, I think that what’s going to happen is that we will have lots of agents either working for us or representing us in many interactions. And so as a result, probably agents will be making a bunch of decisions. And I think of agents as actually not a special thing. Agents are just kind of like fancy software. And we also want to — you know, right now, we don’t necessarily live in a world like this — but ideally, we also want software to be fairly understandable. Like, I want to know what — you know — why is Twitter giving me all of these really annoying tweets? Can I change that? Like, I would really love to be able to change that. Why is it making these decisions?
KANJUN QIU:
And so I actually would love a world where like, yeah, all of the software in my life is understandable and explicable, which I think we’re very quickly barreling towards. You know, Claude Code writing entire code bases, being able to like assess code bases and then answer questions. I want to know what decisions they’re making and why they result in those decisions. And I want to know if an agent is representing me and like kind of making decisions on my behalf, I want to have a fair amount of control over those decisions. And I want to have a fair amount of trust that I can trust this agent to make those decisions. And I want that trust in a deep way. And I want everyone else to be able to trust in a deep way.
KANJUN QIU:
Not just like, oh, I’ve been fooled into trusting this as a consumer who’s not very well informed, but rather like, no, deeply. Like the system is set up such that it is deeply trustworthy. So that might mean something like — at Imbue we’re really big proponents of open source. A lot of the stuff we’re shipping this year is going to be open source, or what we call common source. This idea that in the future we may actually want a lot of our software to share common source code that’s open that we then remix for ourselves or for our groups of each other. I think this is very possible with what’s becoming possible with coding agents.
KANJUN QIU:
So open source is an example of more systemic trust where if a system is open source, like Linux, people like — we can use collective intelligence. Like various people will find bugs in Linux and find security vulnerabilities. And as a result you can trust open source software more because of how it’s set up, not because someone kind of like convinced you that they were a trustworthy entity. And I think it’s important to think about agents at this like systemic level. How might we systemically set up these agents so that they can actually be inherently trustworthy, not just because a company maybe is incentivized to make them trustworthy so you keep using it.
REID:
Most technologists tend toward an individualist libertarian, in example, like how do I validate? But the point is, is of the more than 8 billion people, very few even have the conceptual infrastructure, let alone time, resources to validate it themselves. So the “I validate” basically doesn’t really work. But the we validate and experts validate or commonly validated — e.g., lots of people can look at this and we have a way of surfacing, oh, there’s an issue here. But part of the reason why we have second opinions, we have doctors talk to each other, is they go, well, okay, are we functioning this well? Now, what’s your thought about how this becomes part of the AI landscape?
KANJUN QIU:
I love how you put that, that individual libertarianism doesn’t fully work and that we need like collective forms of validation.
REID:
So that was amplifying your point.
KANJUN QIU:
Yes, it was a really good synthesis and actually captures some of my issue with some of how tech is thinking about these problems and these questions. Like AI is a very societal question. So how do we — I think your question is how do we build systems that like, what is needed to have systems that can be collectively validated and trustworthy? Some of the things that we’re doing at Imbue are — there’s work around openness. So we think of our mission as democratize power and do that by helping to facilitate an open ecosystem of software and agents and data. And so there’s an openness piece and there’s a kind of understandability piece and there’s a kind of like control/modifiability piece.
KANJUN QIU:
And openness enables both understandability and control, but you also have to have additional tools on top of the openness. It just doesn’t give those things by default. On the openness piece, we are trying to figure out — and hopefully other people will do this as well — trying to figure out how do we cultivate a good ecosystem around open models, open software, open agents, and maybe try to overcome the fact that traditionally open source businesses have been difficult businesses. I think that’s one of the great deterrents in our current incentive structure of making things open. However, I also think because software is basically becoming free to build, that there’s a great economic pressure toward openness. Recently I was talking to someone, he built a new product in four days.
KANJUN QIU:
He was like, I can’t really justify charging for this, so I think I’m just going to open source it because someone else is going to replicate it. I think there are great forces toward openness and from our perspective, there’s some additional work needed to maybe enable business models around open source. There are a lot of open source licenses where you can’t really discriminate between personal use and company use. Maybe there’s an easier license or better norms or something like that. On explainability, some of our work at Imbue is about how do you, like — is it possible to ship software that can explain itself to you? Like, yeah, it seems very feasible. Maybe all software ships with something that explains itself to you and like, maybe…
KANJUN QIU:
And then on modifiability — on explainability — there’s other pieces that we’re doing where we have a lab called Feature Lab that is doing foundational deep learning theory. And that theory is really aimed at understanding can we make these systems fully engineerable rather than, you know, kind of probabilistic in the way that they are. Is there like a mathematical foundation for deep learning that can allow that? And if that is possible, then it’s potentially possible to create these systems with a lot less resources. And so that would allow more proliferation of models and like base models and also kind of like the training and guidance needed for them to like be good at instruction following, et cetera. Then on modifiability, I think we’re just going to kind of mostly get that for free. Like people want to modify software. You see a lot of people already like making a ton of stuff, and so I think we’re going to get that for free.
KANJUN QIU:
There’s one more piece on openness, which is about data. So we talk about open software, open agents, and open data. And I think for data there’s like a — you know, I talked about local data, how like agents that have access to our local file system are actually really powerful and kind of like can work with our local data, like Claude Code or Codex CLI. Like we really like these local models, and I actually think we might see — like Imbue is going to invest in an ecosystem of more local tooling that can work with your local data, maybe work with data in your Dropbox so it’s like backed up, but it’s still yours.
KANJUN QIU:
And so I think a better-supported ecosystem around things that like work locally on your computer — agents that work locally on your computer and don’t necessarily like — they’re not kind of black boxes in the cloud that you can’t really change — I think that’s really interesting. And then the second piece around data — so the local data is maybe more private or stuff I want to work with — there’s a second piece around data which is about public data. So earlier I mentioned that quality information really shapes our sense of reality and decisions we make. And there’s a lot of public data out there right now that maybe is not necessarily the information that we’d want to see. And we also lose a lot of data because it’s stored in a server somewhere.
KANJUN QIU:
But I think there’s something really interesting in kind of like a public data backbone for AI agents to work with the Internet. And I think Bluesky and their AT Protocol is an interesting kind of first take at this where basically the way it works is all of the posts on Bluesky are in this public data repository and you as a developer can make any interface you want. And so now you can kind of remix like different pieces of software on top of this public data. And I think that there’s an opportunity to have a lot more like public commons data — data as a public good — that maybe hearkens back to like the old Internet, ’90s Internet, and maybe away from this like Web 2.0 centralization and private servers.
KANJUN QIU:
So all of that to answer — like those are some components of things that I could see helping us become systemically more set up to have trustworthy agents that work for us.
ARIA:
I love that you’re bringing up public data. Just — I’m like real deep on the YIMBY Twitter threads. And especially in New York, there’s like big questions about cars versus Citi Bike. And so someone was spouting some nonsense about Citi Bike, and someone immediately was like, well, I can access the data in New York City on every single Citi Bike station — how many bikes were dropped off, how many bikes were picked up. Oh, someone left that station at 2:59, and I know that 50 people used it that day. And when people park, it’s only five. And so one of the things I’m excited about for public data is obviously the vast majority of people are going to do nothing, are not going to build an agent, are not going to look at public data.
ARIA:
But if we can get like — first of all, there’s so much government data out there that someone who now can utilize that can do amazing things with, you know, Citi Bike and more. And there’s even more that we should categorize. Like with our new abilities, like we could supercharge so many government services if we actually had the data instead of it living in PDFs in like the, you know, middle of DHS files that no one ever sees. If we can bring it to light, like we can do so much more.
KANJUN QIU:
Yeah, I think this is super interesting. One example recently is the UK added a tax on pubs or something like that, and somebody made a map of all pubs that would be affected. And that helps all of these local business owners figure out like how does this tax affect me, et cetera. Similarly, we had the thought when Trump defunded a lot of projects, oh, maybe it would be helpful to like make a list of all these projects and like do that programmatically and make it so that it would be easy for people to collectively fund them. I think there’s something really interesting here about data and specifically what you said about kind of a person being able to correct the like public discourse.
ARIA:
Absolutely.
KANJUN QIU:
Because yeah, in the theme of this podcast, Possible — like right now we have an Internet that has a lot of fake news on it and like false information. And in theory, with language models, we should be able to — with language models and a better public data record system — question mark — we should be able to have a system that tends toward truth and tends toward people being able to have more truthful understanding of reality. And so that’s like starting to become possible. And I’m like super excited about that path.
REID:
Well, I think part of what you need — and I agree that like lots of data being more publicly and generally available, it’s good — as you just mentioned, there’s a lot of that which is not — is worse than garbage on the Internet because it’s deliberately misleading.
ARIA:
Right. Garbage is just neutral.
REID:
Exactly. So a lot of the anti-vax stuff is in this vector. But another thing that is important is how do you weave the logic of understanding this data together? In addition to the data layer for making a more humanist set of things, what are the things that you think need to be done on the reasoning side and on the understanding of the quality of data side?
KANJUN QIU:
This is a great question and is actually something that if somebody has a project that is trying to work on excavating truth from a lot of data streams, I would love to fund that project or have them work on it at Imbue. I am not sure what the answer is. I think there needs to be really thoughtfully designed — like a really thoughtfully designed layer kind of on top of the Internet. An example of a problem that cropped up recently is there was some fake ChatGPT-generated news that was published and then another model cited that, like something else cited that news and something else cited that news and then now it became real and then the models thought they were real. And you’re like, okay, that’s not good.
KANJUN QIU:
Like we need some way to like deeply investigate this whole chain of source citations and to understand like is it actually real? And so maybe that looks like agents that are investigative agents that then report their findings just like a scientific meta-review into some central database — like, I guess decentralized, but like central database.
REID:
Centrally accessed. Even if decentralized in terms of where all the data is and the links between.
KANJUN QIU:
That’s right. And so yeah, maybe that’s what that looks like. I’m not quite sure what works, but there’s something interesting in the scientific kind of process.
REID:
Yes.
KANJUN QIU:
That’s been developed, and I think these are actually like social process innovations plus technology to support those social processes. Not quite sure what it looks like yet.
REID:
One of the frequent things that I find challenging — and you know RFK Jr. is a poster child for this — which is like people misunderstand to say, well science, you know, that broke there. And you’re like, yes, science is a continually learning mechanism with reproducibility. The answer is not: it’s always right. The answer is: it’s on a trend towards being right in the process because like most people are not good at understanding statistical argumentation. What are the ways that we get to common knowledge, right — like common sourcing? What are some of the ways that if we’re steering AI systems to get to this — and I deliberately use the term — possible future, what are some of the things you think we need to be doing as technologists and a society in order to get there?
KANJUN QIU:
Yeah, on the like, possible futures, I think this is one of the most — my most optimistic views of what progress looks like with AI is that AI by its nature synthesizes a lot of conflicting information to present to us what it seems to understand as the like — what its best understanding of the truth is, if it’s trained well. I actually have a maybe hot take here where like I think one of the most important things — I think there’s like a systems thing and then there’s a moral thing. So there’s a systems thing which is like Anthropic making AI systems that are truthful, as truthful as possible. And I think Anthropic culturally really cares about that. And like I love that they exist.
KANJUN QIU:
I think there’s a moral thing, which is as technologists, we build incredibly powerful things that change people’s lives and affect their day-to-day, but we don’t have a variant of the like Hippocratic oath. And like something I’ve been kind of noodling on is like, what would the Ten Commandments for technologists look like? One example is I’ve been thinking about why some things feel more fake than others. For example, like if we’re scrolling Instagram at 3 a.m., that feels fake. But like watching a great movie until 3 a.m. feels kind of more meaningful, more real. Or if we’re eating processed food, it feels kind of fake. But like if we’re eating really nutritious food, it feels like more real. Or if we see someone beautiful in a beauty magazine but they’ve been photoshopped, that feels fake.
KANJUN QIU:
But like in real life, if they’re like really vibrant, that feels real. What’s going on? And I think there’s this thing that can happen to humans as we get more powerful technology that I’m calling sensory hijacking. So technology hijacks these sensors that we have to determine actually how to live really meaningful and healthy lives. We have really good sensors for that, but they’re very, very hackable. And so these systems that capture our attention, hijack our attention, are hijacking some of these sensors that exist for that — you know, that determining a meaningful life. And so like one example of a commandment might be like thou shalt not hijack other people’s sensors. Another one might be like thou shalt not build systems that propagate false, you know, reality. Or like that’s another kind of sensory hijacking. You’re like giving people wrong information on purpose.
ARIA:
When the rubber hits the road, how do you make this happen? Is it a culture change? Is it government intervention? Is it all of the heads of the AI labs getting together and being in conversation with each other? I think so many people — sort of in this room, certainly, but in our position — want the better future. They want to drive toward these sort of humanistic goals. But we don’t want to stop technological progress because we see how it makes people’s lives better in so many ways. There’s also geopolitical implications. But I think people are still struggling for then like how do we cross that chasm? Because you’re talking about some of the harms, but how do we get to the other side?
KANJUN QIU:
Yeah, I think some of the things I talked about — like I think it requires a lot of invention actually. Some invention and some kind of internalization of morality and excavation of what is moral technology. Invention at the systems level. How do we invent systems that actually do tend toward truth? How do we invent systems that allow for more openness and trustworthiness, like software systems, digital systems? How do we invent economic systems that allow for more agency and more authorship, especially as AI becomes more powerful and is able to do more of people’s work? Like this is an open question I have. I don’t really know how that’s going to shake out. So there’s I think a lot of invention required over the next few decades to get here.
KANJUN QIU:
And I think it’s — if I knew what exactly to invent, I would have done it already.
REID:
(laughs) Yeah, well, it’s a process.
KANJUN QIU:
It’s a process. That’s right. It’s a lot of what we’re trying to figure out at Imbue and why we’re such a strange company: trying to invent these systems that allow for a future that is like systemically more robust and more trustworthy and more agentic and not just, you know, not fragile.
REID:
Of technologists, you are at the far edge of philosophical sophistication, right? There are many technologists who are not coming to a growing common perspective on morality as also just like common knowledge or common source. There’s kind of like the — okay, how do we get to that common morality, and how does AI play into that?
KANJUN QIU:
It feels like as humans we’re always trying to figure out what leads to a more like meaningful life. Like why do we exist? What is a good life with the amount of time we have on this planet? And like some of us figure it out early, like you. Some of us figure it out late, like me. You know.
REID:
(laughs) I’m not quite sure of this, but that’s okay.
KANJUN QIU:
It’s taken me a long time to be like, oh, finally I can do the things I want. I don’t just have to do what people expect of me. There’s something really strange and interesting about meaning, agency/authorship, and being controlled. There’s like some dynamic here. I’m gonna try to point at it. I think we are controlled often by the world that we live in. Not necessarily badly — we don’t murder other people — it’s a form of being controlled. I think we do internalize a lot of maybe like — I’m gonna use this term, it’s not quite right — oppressive or controlling narratives. And we do exist in some kind of controlling systems. Like my phone is a controlling system.
KANJUN QIU:
And so I think if I were to answer like what is the human good or like how do we build a positive future for humanity, it’s something like we want a society that’s able to continue to make progress on what a meaningful life is. I think there is real kind of like thinking and quote unquote progress to be made there. I think we understand a lot more about it today because we understand our world and ourselves and our minds and what makes us human and why that is valuable. So I think we want a society and systems that like let us continue to make that progress. And I, at the same time —
KANJUN QIU:
So that’s at the like social level, collective level, and then at the same time at the individual level that like protects us from negative things — that’s like negative liberty — and also kind of like induces positive liberty, positive freedom. Like induces agency and sense of authorship and like protects us from things, systems that might control us or tell us false things. And so the combination of these two I think in the context of technology is some of what we’ve talked about today, but not comprehensive. Like in the context of technology, being protected from some of the negative control might be like being protected from a system that might manipulate us, being protected from easily ingesting false information, being able to trust agents that operate on our behalf because they’re systemically trustworthy. Those are like protective stances.
KANJUN QIU:
And then there’s like a positive freedom aspect of it. A lot of what you do, Reid, I think to kind of like give people examples of what’s possible. They like — it’s really helpful to see what somebody else did. Like when you wrote this book, Superagency, I was like, oh my God, I can do this. Whoa, that’s so cool. And it like, actually unlocked a lot of writing for me. That was really cool.
REID:
Thank you. That is great. Yes, yes.
KANJUN QIU:
So like more examples of people like doing stuff that’s possible and then also updating some of our social narratives that kind of keep us operating in like, oh, we’re just this way or that way, and business as usual. My co-founder has a really interesting thought experiment, which is let’s imagine that 10,000 years from now there are some human descendants who live on a faraway planet orbiting a faraway star and they see Earth through a telescope. In the case where they have forgotten where they came from, Earth is just another planet. It has no meaning to them. And that’s actually really interesting. In the case where they have remembered where they come from, Earth has this deep meaning to them. And so it actually says something interesting about meaning.
KANJUN QIU:
Like what is meaning to people? There is something in this thought experiment that I don’t quite understand that says something about meaning.
ARIA:
Well, I was just thinking about a hypothetical where you had your personal AI agent and we’ve solved the misalignment. We’ve solved the trust issue. Like you trust it. It has your, you know, best interests at heart. And so it, you know, if you’re feeling down, it knows you. So maybe it tells you about meditation or maybe it tells you to go to church this Sunday. Or maybe it tells you to get a new job that has more purpose. You know, there’s some universal ones and, you know, some — go get your flu shot. It’s gonna be a really bad flu season. And so like it does all the right things. And so I guess I have two questions.
ARIA:
One is like is this the utopia where everyone has this personal AI agent that we trust and is aligned and we are on board? And then you made me think, how does that — like the good news is that helping others and having a purpose is actually good for you — but is there some misalignment because it’s good for you but it’s not necessarily looking out for society? Is there a trade-off between the personal and the collective if we have this magical AI agent?
KANJUN QIU:
I think this is a really interesting question and kind of speaks to the amount of power someone who trains these models has, which is you know you can think of an agent that has both your values and societal values at heart. And maybe the agent that has societal values thinks, okay, what is best for society? Well, if you buy an SUV, here are the trade-offs for society. However, you value freedom and personal liberation. And so maybe you want an SUV. So an agent that really strongly considers both might say you might want an SUV, but know that the trade-off is that you’re making things worse for other people. However, an agent that doesn’t consider that and only considers your values might be like you should get an SUV. Like this is best for you.
KANJUN QIU:
And so I think I’m confused about like — I personally would prefer the agent that informs me of the trade-offs of my decisions. I think that’s good and kind of like helps me become more collectively oriented. But not all agents are going to do that for sure.
REID:
Well, and part of the challenge when it gets to human nature is that many human beings prefer certain kinds of self-deception, right? They prefer to — as opposed to saying, hey, when I get this SUV, I’m actually having the following kinds of impact on the world. They go, no, no, no, that’s, that’s —
KANJUN QIU:
I don’t want to know that. I’m not gonna buy this.
ARIA:
I’m a good parent. I’m keeping my kids safe. And my first identity is as a protector of my children. So it’s really easy to just say it’s totally fine and it’s what I should do as a moral human.
KANJUN QIU:
Exactly.
REID:
Because my hope is that with AI and its ability — it is the best learning tool that we’ve constructed in human history — is that it helps us all become somewhat more truth-seekers. Yes, it’s a definite hope. It’s a definite, trying to steer in that direction. But it’s a very complicated path.
KANJUN QIU:
Yeah, that’s my hope also. Like, you know, I think maybe that a lot of us lose faith in education because we’re not really met. We’re either overly bored or overly stimulated — it’s like too much on either side — and like these systems can meet us where we are. So maybe that will help us get closer to being truth-seeking. I think humans do want to — in our nature — understand what’s going on and that like we only lose touch with that when we’re really distracted by all this stimulus or like distracted by old hurts. And so maybe like, you know, part of it is like AI being better tutors, helping us understand things better.
KANJUN QIU:
Maybe like part of it is like everyone meditates with this new method that’s really efficient and like we’ll all see the true nature of our experience as like a field of compassion and love and then that will solve all the world’s problems. I think it’ll be like a slow progression toward something like that, hopefully.
REID:
And just so that our listeners don’t think it’s unduly optimistic or Pollyanna-ish, the answer you just gave actually in fact is deeply within the Western philosophical tradition because it’s Socrates, and it’s deeply within the Eastern tradition because it’s Buddha.
KANJUN QIU:
It’s true.
REID:
And both of those two figures would say the thing you said was exactly right. And we just need to help people on the dialectic toward it. Now I think there are complexities to it, yes. But it’s a good — it’s not an unconsidered objective that actually in fact leads us into better places.
KANJUN QIU:
Yes, I love that. Yeah, we have an opportunity to kind of really lean into the Socratic method, Socrates, and really lean into like, what Buddha discovered about the fundamental human experience. And maybe that would teach us something.
ARIA:
So we’ve been talking a lot about sort of philosophical power in the future and how do we point toward this humanistic future. But economic power is going to be critically important, and how do creative people retain those economic rights? How do — do people even have jobs in the future? And so what do you think needs to change about ownership and monetization in the future as we think about AI? You run a research lab and you need some feedback signals. And so what are the — what are the things that we should use day to day as we sort of hurtle toward this future when we think about the economic piece of it?
KANJUN QIU:
I think I personally am most confused about the economic aspect of this entire situation. I have this model of capitalism that I’m not sure is correct, but it’s something like there are two core mechanisms. One core mechanism is the markets, which is like a coordination and resource allocation mechanism. And the markets are amazing because they allow people who are like more self-interested — like they tap into human self-interest — and allow that self-interest to be super beneficial to other people. And then there’s a second mechanism which is like the idea of return on capital and kind of being able to invest in something and get more returns on capital. And I think when people talk about capitalism, they often combine these two mechanisms.
KANJUN QIU:
They’re like, oh, capitalism is like this whole thing that like, you know, people invest and like get money back and like — but reading Adam Smith, Adam Smith is like only talking about the first mechanism. And he’s actually very concerned about like monopolies and rent-seeking and the kind of perpetual growth machine that we currently live in. I think that the perpetual growth machine combined with AI that’s really, really smart causes serious problems. I could see AIs that do a lot of kind of hijacking or hacking of our existing system to accumulate capital or resources. So like people who get really good at high-frequency trading — I was a high-frequency trader a long time ago — and like AIs can definitely do this well. But is high-frequency trading actually connected to real production, like real production value?
KANJUN QIU:
And that was one of Adam Smith’s other concerns, is like financial returns that are disconnected from real production value. And so I think we currently live in this economy where like a lot of our financial returns are disconnected from real production. It’s like kind of hard to evaluate exactly how or in what ways they’re disconnected. And so for me, the only part of it I know what to do about is that I think I want to see if we can make software go from a renter’s economy to an owner’s economy. Right now we’re in a renter’s economy of software. Pretty soon I think we can be in an owner’s economy of software. Seems possible. Seems like, you know, the production costs are getting really low, and so that’ll change some things.
KANJUN QIU:
But I’m not sure it’ll address much of the problem. You know, I don’t know how to think about the rest of like, yeah, how do people get paid? Who gets paid for what? Who makes the money? Where does the money go? Does the money accumulate? Does it distribute for some other reason? Like maybe people are super creative and then somehow we end up with like Substack everywhere for everything.
REID:
So now to rapid fire, which is — is there a movie, song, or book that fills you with optimism for the future?
KANJUN QIU:
I love The Beginning of Infinity by David Deutsch. I think it is one of the deepest excavations of what progress is that I’ve ever read. I read it in 2020, and before that I was like, I don’t know if progress is real. There’s this kind of meme among historians that history is cyclical. What’s the point? What is progress? I can see that our standard of living has improved. Is that progress? It seems like progress. But like what does it actually mean? The idea he puts forward is that the construction and layering on of good explanations is what results in progress over time. And so a good explanation is an explanation that explains the phenomena and has reach. Like it can also explain other phenomena.
KANJUN QIU:
For example, he gives a really interesting example. Like the explanation that the sun rises because Helios pulls it in a chariot every day is like — so his first criterion is that it’s hard to vary. Well, you can vary a lot of things about Helios: his hair color, his eye color, how tall he is, what the chariot looks like. And the second thing is that it doesn’t have any reach. Like it can’t explain anything else in the world. Whereas the axial-tilt theory explaining why the sun is where it is in the sky is very hard to vary and it has a lot of reach. It also explains seasons and it also explains like why seasons are different in different hemispheres.
KANJUN QIU:
So the idea that the kind of layering of good explanations is a sign of progress I think is actually the most convincing argument I’ve heard for what progress is. And this book fills me with optimism. You know, even if we fulfill Maslow’s hierarchy, like what cultural values make sense and like how — how do you — there must be some like more fundamental underlying explanation about why love and like kind of like altruism is better than hate and anger. But like what is that explanation? And like I think his argument would be the excavation of these good explanations is what leads us to these progressions in moral philosophy and in economics and these other things.
ARIA:
What is a question that you wish people would ask you more often?
KANJUN QIU:
I think a thing I don’t talk very much about because I am like an AI person, quote unquote, so I don’t have very many opportunities to talk about it is I actually have systematically removed my anxiety over the past 10 years. And like I used to be a super anxious person, and being a founder was really hard. And like I had all these voices in my head and they were really critical. I have Asian parents and Asian culture and like very critical voices in my head from my childhood. And I think people assume that anxiety is just something you have to deal with and the voices in your head are just like something you have to learn to befriend. But it’s false. I think that trauma is basically —
KANJUN QIU:
Or like these voices are basically overfit models. And as we grow up, our data distribution changes. And what we have to do is go access those models and like systematically update them by giving them new data about our lives. And so that’s what I’ve done with all of these different modalities. And it’s like really effective.
ARIA:
Do you have advice for someone who’s experiencing that and wants to fix it?
REID:
Say?
ARIA:
(laughs) No, not me.
REID:
Asking for a friend?
ARIA:
I don’t have anxiety, so I don’t know about it, but I know it’s crippling for people.
KANJUN QIU:
(laughs) Asking for a friend.
ARIA:
(laughs) Truly.
KANJUN QIU:
Truly? You don’t have anxiety?
ARIA:
No. None.
KANJUN QIU:
That’s amazing and unsurprising. That makes so much sense.
REID:
All right, so where do you see progress or momentum outside of AI that inspires you?
KANJUN QIU:
One of my best friends works in longevity. And I think like the medical sciences and what AI is doing — and not even AI, like outside of AI, what we’re starting to learn about how our bodies work and like turning things like drug discovery or vitrification of organs into like freezing of organs so that they can be preserved for organ donation — like turning them into engineering problems — is just like super interesting. And I’m really excited for the next five to ten years. I think we’ll learn a lot here.
ARIA:
Can you leave us with a final thought on what you think is possible to achieve in the next 15 years if everything breaks humanity’s way? And what’s the first step to get there?
KANJUN QIU:
I think it’s possible to have a digital world that we feel like we can change it — like anything in our house or anything in our lives or anything in our kitchen. Like maybe our digital world feels more like our kitchen, where we know how to make food and have the tools for it. We can rearrange our kitchen. We can design it exactly how we want it to. This idea is not unique. It’s from Glenn McDonald on our team. I think it’s very possible for all of us to feel empowered in the digital world in the way we feel empowered in our kitchens and for that environment to kind of nourish us the way our kitchens can nourish us.
REID:
As always with great conversation, Kanjun, I have questions for another 10 hours. So one hour leads to ten. But I look forward to the next one.
KANJUN QIU:
Me too. Thank you both.
ARIA:
Thanks so much.
KANJUN QIU:
This was really fun.
REID:
Possible is produced by Palette Media. It’s hosted by Aria Finger and me, Reid Hoffman. Our showrunner is Shaun Young. Possible is produced by Thanasi Dilos, Katie Sanders, Spencer Strasmore, Yimu Xiu, Aman Suri, Lexxi Kivvens, Danny Garrison, Trent Barboza, and Tafadzwa Nemarundwe.
ARIA:
Special thanks to Surya Yalamanchili, Saida Sapieva, Ian Alas, Greg Beato, Parth Patil, and Ben Relles. And a big thanks to James Campsie, Brandy Hagle, Ekkta Bathija, Erik Pavia, Ben Casnocha and the Village Global team.