学术研究属于有野心的人——MIT 的 Alex Zhang

Last call for regular tickets for AI Engineer NYC! As an exclusive for Latent Space subscribers, the first 30 of you can take a 30% off code if it helps - for new tickets only, no refunds! See you in 2 weeks!

While we tend to cover industry on the pod, every so often we celebrate a clearly emerging superstar PhD. In 2024 we featured Shunyu Yao, who went on to build Operator at OpenAI and is now Chief AI Scientist of Tencent. In 2025 we featured Jack Morris, who went on to cofound Engram at $600m and is now a leading voice on continual learning.

This year we are proud to feature the work of Alex Zhang of MIT.

From GPU kernels and KernelBench to Recursive Language Models, Mismanaged Geniuses, and massive multi-agent swarms, Alex Zhang is exploring how much capability we’re leaving on the table by wrapping increasingly powerful models in primitive systems.

RLMs took over the timeline early this year:

and an RLM based harness was the first to ~solve ARC-AGI-3 before OpenAI’s Astra:

and is even today, influencing new research that has more extreme implications than RLMs:

We go deep on GPU Mode and AI-written kernels, research taste and why academics should take bets industry labs won’t, GEV and alternatives to the standard autoregressive language model, and the idea of harnesses as compositional generalizers. Alex explains RLMs, context offloading, programmatic subagent calling, Prime Agent, persistent subagents, and why the “language model” of the future may actually be an invisible swarm of agents underneath a simple interface. We also discuss OpenAI’s massive agent experiments, Kimi swarms, open-ended research at Sakana AI, speculative programmatic tool calling, capability overhang, Neuralese, and where Alex thinks the next big research opportunities may lie.

We discuss:

- Why AI-generated GPU kernels still leave substantial room for human expertise

- How one expert insight can potentially replace enormous amounts of brute-force token search

- Why PhD students should take research bets that initially look trivial, weird, or pointless

- What SWE-bench, RLMs, ReAct, and Quiet-STaR reveal about research taste

- GEV and why a language model does not have to mean an autoregressive text-to-text decoder

- Why Claude Code, Codex, and Pi are structurally more similar than they look

- How harness design can improve compositional generalization across tasks and domains

- RLMs: context offloading, code execution, recursive subagents, and shared memory

- Prime Agent, continual harnesses, and persistent agent-to-agent communication

- Why the model you query in the future may secretly be an entire swarm or scaffold

- OpenAI’s 10,000-agent experiment, 130B output tokens, and ~$40M-equivalent problem solving

- Why much of an agent swarm may be wasted search — and why convergence is still hard

- Kimi versus OpenAI and different approaches to multi-agent systems

- Open-endedness, Sakana AI, and finding hidden gems in enormous amounts of generated work

- Why current frontier models may already have a large capability overhang

- Speculative programmatic tool calling and overlapping tool execution with generation

- Whether English, code, or an entirely new “Neuralese” constrains how models reason

- AI for science, fast-moving benchmarks, and how Alex chooses what research problems to bet on

Alex Zhang

- Website: alexzhang13.github.io

- X: @a1zhang

Timestamps

00:00:00 Introduction

00:00:49 GPU Mode, KernelBench, and AI-Written Kernels

00:07:38 Human Expertise vs. Brute-Force AI Search

00:13:20 Research Taste and Taking Big Bets

00:19:28 GEV and Rethinking the Language Model

00:29:03 Video Game Agents and the Harness Problem

00:31:01 Why Claude Code, Codex, and Pi Are So Similar

00:36:42 Harnesses as Compositional Generalizers

00:44:24 RLMs Explained

00:52:01 Prime Agent and Persistent Subagents

00:57:41 RLMs in the Wild

01:00:30 OpenAI Swarms and the Future of Language Models

01:07:26 Open-Endedness and Sakana AI

01:15:52 Kimi vs. OpenAI Agent Swarms

01:20:06 Capability Overhang and Speculative Tool Calling

01:28:19 Neuralese, Future Research, and AI for Science

Transcript

Introduction: Alex Zhang, RLMs, and GPU Mode

Swyx [00:00:00]: All right, we’re here in the studio with Alex Zhang, I guess most famously of, RLMs, but you have a few other affiliations. Welcome to the show.

Alex Zhang [00:00:12]: Yeah. Thank you for having me.

Swyx [00:00:13]: Yeah. I guess GPU Mode as well?

Alex Zhang [00:00:15]: Yes, GPU Mode as well.

Swyx [00:00:16]: You were shepherded in by Mark Saroufim. Not everyone gets that kind of welcome.

Alex Zhang [00:00:19]: Yep. Yeah. Yeah. I’m very close to all the people in GPU Mode, so yeah

Swyx [00:00:23]: Yeah

Alex Zhang [00:00:23]: We often end up working together in various capacities, like even beyond just GPU Mode itself, so.

Swyx [00:00:29]: Yeah. Can we explain, so people who are not that close Don’t know about this. It’s just a-- it’s a Discord. It used to be focused on, I guess, CUDA Mode, and then generalized a little bit. it was started by Mark.

Alex Zhang [00:00:41]: Yep.

Swyx [00:00:42]: It was basically like. To me, it’s like the hiring pipeline of the PyTorch team.

Swyx [00:00:45]: And then you left PyTorch.

Alex Zhang [00:00:47]: Yep.

Alex Zhang [00:00:49]: Yeah. So it used to be, I think, it started actually around when I was in college, in like 2023. I think it was started by Mark, Andreas, and Jeremy Howard. The original premise was just like, it was a GPU-- or it was a Discord dedicated to learning how to write GPU kernels, and they had, like, lectures. That was basically the extent of it. and I got interested in it because I was writing GPU kernels. It was actually out of, like, pure chance. I was interning at Snapchat at the time, and

From CUDA Mode to GPU Mode

Swyx [00:01:22]: Rexis.

Alex Zhang [00:01:23]: Yeah, I was very bored with Rexis. So, they had a project where, like, they were interested in writing. It was this paper called Infinite Attention. It was like a Google paper.

Swyx [00:01:35]: Yes, we’ve covered it on Paper Club.

Alex Zhang [00:01:36]: Yes, yeah. So I was interested in whether or not you could write specialized kernels for it at Snapchat. it didn’t. Nothing really came of it, but I joined GPU Mode. At the time, it was called CUDA Mode, I think for, like, legal reasons or something, they changed the name. But I met Mark, I met Matei, I met a bunch of other people that were very involved in the community. And then Mark had pitched this idea called Popcorn, which was now what you see as the leaderboard today. But the general idea was like. I think all of us had this, like, intuition that GPU programming is, like, very similar to if you guys have done, like, competitive programming. It’s a, it’s a not. I don’t mean to say, like, they’re transferable skills.

Popcorn, KernelBench, and Automating GPU Kernels

Swyx [00:02:17]: You have constraints. You code golf a little bit.

Alex Zhang [00:02:19]: Yep.

Swyx [00:02:19]: Yeah.

Alex Zhang [00:02:19]: Yeah. And there’s like. There’s actually a surprisingly small space of optimizations that people do. and there’s actually not that many kernels per se that people are interested in optimizing. And so we kind of had this thought that, like, if you had enough data, like, in the same way that Codeforces, there’s millions of problems. If you could do this with GPU code, like, you could scale and automate kind of GPU kernel development, which for researchers is a huge deal. ‘Cause I think one of the bigger bottlenecks. Like, if you look at, like, Mamba, for example, like, they release the paper with kernels because otherwise, like, you can’t really use it in any meaningful way? And not everyone has, like, a Tri Dao on their team. So we’re very interested in this. KernelBench kind of spawned from that too, of like, can we get LLMs to automate, GPU kernel code? And I think that was like a. It was a very fun time. It was like between college and my PhD, and yeah, I had a really pleasant time doing stuff with GPU Mode. Now I kind of just help with the lectures sometimes. I’m not as involved, and I think in general, like, we don’t have as many competitions as we used to. But, yeah, I still keep in touch a lot with everyone there.

Swyx [00:03:27]: Is there a friendly rivalry? Because I think the previous community that used to do this was like MLSys, MLPerf

Alex Zhang [00:03:32]: Yeah

Swyx [00:03:32]: Kind of thing. Is there a friendly rivalry? Is this like just new generation MLPerf, or what’s going on?

Alex Zhang [00:03:38]: The nice thing about GPU Mode is that it is also a community in the sense that, like, a lot of the lectures are very easy enough for a beginner to follow and ask questions and things like that. And, like, the competitions are, like, somewhat, not secondary, but, like, you can participate in them to learn. I think with a lot of, like, MLSys, MLPerf kind of benchmarks, like, for the most part, like, only serious labs and companies participate. Like, seriously in them, at least. That was my understanding of it. I could be wrong. But I think also beyond GPU Mode now, one thing that has been really exciting is there’s a lot more websites and, like, people that work on hosting competitions. Like, I think there’s this.

Alex Zhang [00:04:21]: I think there’s this website called, like, LeetGPU or something, and it’s like leet code for GPU problems.

Swyx [00:04:26]: Wow.

Alex Zhang [00:04:26]: There’s, like, other ones that I’ve. Like, we’ve, we’ve seen. Like, there’s many that have kind of spawned and, like, talked on GPU Mode, and like, it’s very exciting in the sense that I think GPU programming used to be super niche, like when I was interested in it. And the only reason I got interested in it was Tri Dao gave a talk at Princeton because he was, applying for faculty. he is faculty there now, but I listened to his talk on FlashAttention in like 2023, and I was like, “Wow, this is like the coolest thing ever.”

Alex Zhang [00:04:56]: And I was like, “This is like. This is what everyone should be working on.” I guess, like, vLLM and stuff had come out too, and it was like, “Oh, we should be writing kernels.” But now it’s like, everyone writes kernels. Like, everyone. It’s, it’s. I think it’s actually almost saturated in some sense, as a field.

AI-Written Kernels and the Verification Gap

Vibhu [00:05:11]: Any interesting takes for people that wanna get into it? So I think one of the biggest news is GPT-5.6 Wrote more efficient kernels, so Terra and, Luna could be 80% cheaper.

Alex Zhang [00:05:24]: Yeah.

Vibhu [00:05:25]: And then we’ve seen other competitions where people are, like, setting records, and they’re like, “We’re doing some auto research loop,” and these are people that don’t have a background

Alex Zhang [00:05:34]: Yep

Vibhu [00:05:34]: In any kernel writing, right?

Alex Zhang [00:05:36]: Yeah. So even on the GPU Mode leaderboard, if you look at, like, a lot of the recent problems, almost all the solutions are AI generated. However, you’ll notice on the leaderboard. So there’s this guy named Gauners who’s, like, a very, like, regular member of GPU Mode. We’ve always known for a long time that he’s, like, a super cracked, like, GPU kernel writer. One thing we discovered on this leaderboard is, like, almost ev-- Like, he also used AI to help him with these solutions, but for the most part, like, he helped prompt and move it in certain directions. we found that, like, his kernel was, like, basically the only one in, like, the top 10 that was actually stable in, like, actual, like. end-to-end systems. And it does bring into question, like, it’s not. Like, GPU kernels have a verification problem. Like, we’ve kind of known this. It’s been a problem since KernelBench was released. Like, there’s a lot of reward hacking that goes on. But you al-- Yeah, you also notice, like, the lines of code is a lot smaller, but

Vibhu [00:06:33]: Yeah, I was gonna ask, is that noticeable, or is it just

Alex Zhang [00:06:36]: Yeah, no, it’s, it’s

Vibhu [00:06:36]: Okay

Alex Zhang [00:06:36]: It’s definitely, like, very important, and I think, like, it’s, it’s really interesting that still there’s a lot of alpha in being good at writing GPU kernels.

Vibhu [00:06:44]: Okay, so there is a gap from verifying

Alex Zhang [00:06:45]: There definitely is, yeah. I think, like. And this applies to a lot of AI systems as well. Like, I think, even with the most recent, like, math proofs and stuff, like, it doesn’t necessarily mean mathematicians are obsolete. these companies still hire mathematicians, like, to do, whether it be, like, data labeling work or even just, like, steering the models to solve problems. Like, there is still a lot of alpha in being, like, knowledgeable in these things, so.

Swyx [00:07:12]: Is it just knowledge, or is it also there is just more planning, and is there, an emergent style of planning that works better?

Alex Zhang [00:07:22]: I think it’s, it’s a mix of. Maybe this is what you mean, like intuition for

Swyx [00:07:27]: Something like that

Alex Zhang [00:07:28]: How to solve the problems.

Swyx [00:07:29]: Like, for example, I always diagram my code.

Alex Zhang [00:07:31]: Yeah.

Swyx [00:07:31]: Right?

Alex Zhang [00:07:31]: Yeah.

Swyx [00:07:31]: And then, like, if there’s a part of the diagram I don’t understand, I work until I understand it. Otherwise, I, it’s not allowed.

Alex Zhang [00:07:37]: Yeah.

Swyx [00:07:38]: Yeah.

Alex Zhang [00:07:38]: So I think it’s, like, it’s a mix of those things of, like, the people who work. Like, the people who know how to look at these problems and how to solve them, like, also know how to use AI to do them. Because, like, you’re acting as a very strong verifier. Like, if you are knowled- or if what to do and you are also. Like, I think the thing that we’ve kind of discovered with all these agent swarms and things like this is, like, when you throw enough compute at a problem, you, like, can sufficiently explore solutions to that problem. But oftentimes, like, maybe you can burn, like, 100 billion or a trillion tokens on something, but if you bring in someone who knows something about the problem, they can uncover something for the model that would, like, erase that one trillion token spend. it’s not, it’s not super clear, like, what exactly the trends are here. But I think, like, there are so many problems in the wild still right now that we want to solve, and, like, we can’t afford to just always, throw as much compute as possible at it. Like, there is still an efficiency aspect of all of these things that is super important.

Speed-of-Light Limits, Memory, and Megakernels

Swyx [00:08:41]: Is there, like, a theoretical right answer that you just calculate based on physics, and then you just get close to the physics limit?

Alex Zhang [00:08:49]: Yes. So for GPU kernels, you can compute. It’s actually not that easy to compute sometimes, like, depending on how complex the problem is. Like, for matrix multiplication, it’s very easy to compute, this, like, speed-of-light kind of, estimate of what the fastest kernel can be. And, like, this is also assuming, like, maybe all of your, all your data starts on the CPU, or maybe it starts in DRAM, on the GPU, et cetera. Like, this changes these numbers slightly, but

Swyx [00:09:18]: The transfers and all these things, yeah.

Alex Zhang [00:09:20]: Yeah. But I will say, like, it’s not clear, though, like, in a lot of cases if it’s even possible to hit this theoretical number, if that makes sense. Like, this is assuming, like, perfect overlapping and transfer of data, and, like, there’s maybe some bottleneck that you can’t get around. But often, the kernels are not even close. Like, that we write are not nearly close enough to this number to be, like, meaningful at all.

Swyx [00:09:43]: Yeah. And is it speed that matters? Do you also care about, obviously memory, which

Alex Zhang [00:09:48]: Mm

Swyx [00:09:48]: Feeds into speed? Do you care about power consumption? So one of my, one of our top pods of the year was Geoff Dean, who was like, “Actually, I just tracked the microjoules or, like, the nanojoules, picojoules.”

Alex Zhang [00:09:59]: Yeah, it’s often picojoules today.

Swyx [00:10:01]: Picojoules.

Alex Zhang [00:10:01]: Yeah.

Swyx [00:10:01]: Do you care about that?

Alex Zhang [00:10:03]: So I don’t.

Alex Zhang [00:10:04]: Yeah. I guess maybe I’m not, I’m not as

Swyx [00:10:06]: But everything here is speed, right? Like

Alex Zhang [00:10:07]: Everything here is speed

Swyx [00:10:08]: Nobody’s counting picojoules.

Alex Zhang [00:10:09]: But I-- There’s a caveat here, which is, I think, like, there is speed in the context of a single kernel, and there is speed in the context of a larger problem, like maybe the N10 model. Because, like, one thing to consider, and this is why it’s important to talk about what speed-of-light is referring to, because in these cases for the kernels, like, we always start with everything in, like HBM, for example, right? But you can imagine that, like, an end-to-end like an end-to-end model, what you might wanna do between two layers is, like, you might sacrifice the speed of the first operation to keep things in the cache for the second operation. And, like, these are things that, like, you can’t really get out of, in isolation with, like, these kinds of kernels. And, people call this, like, the fusion or, like, the

Swyx [00:11:01]: Megakernel

Alex Zhang [00:11:02]: Kernel fusion problem. Yeah, or, like, megakernel stuff. And it generally only applies, like, when you are, like, memory-bound in most cases. But this is something that, like, also there is this question of, like, as these models get better, like, should we just be generating like, megakernels? Is that, like, what we want?

Vibhu [00:11:20]: What’s, what’s your take?

Alex Zhang [00:11:21]: I think that this is really difficult because you need the data to do this. And I think, like, I have yet to see an example in the wild of, like, we bootstrap the ability to solve a very difficult class of problems without any examples. and I think, like, the other reason why I think maybe this isn’t that interesting is that at the level of an individual kernel, A, like, they’re not that, they’re not as complex, but B, you’re somewhat confident that there’s not as much structure in a single kernel. But, like, in a megakernel, like, I would be more inclined to believe that, like, a compiler would be better here. Like, some compiler over, like, higher level- Ops makes sense, because in general, like actually, I think mega kernels are very, like the pieces are very composable of like the individual kernels. There’s some areas where you might wanna do like weird fusions and everything, but in general, I think these are cases that like a compiler can probably handle. And there is a company that’s working on this from what I understand that has given some talks on GPU mode as well.

Swyx [00:12:28]: Yeah, I wanna basically cluster all the GPU mode discussions here because obviously there’s other parts

Alex Zhang [00:12:32]: Right.

Swyx [00:12:32]: That we need to move on to.

Vibhu [00:12:33]: I think there is something to plug. You guys do host a lot of really good lectures. They’re all on YouTube. People can follow. And you

Alex Zhang [00:12:39]: Yes

Vibhu [00:12:39]: Lead quite a bit of it. You’re still quite involved.

Alex Zhang [00:12:41]: I used to. sometimes I still do. I think they’re mostly Mark. Mark is the one who usually does them. Matei does sometimes as well, but, yeah, I highly recommend them. They are extremely good resources. Like, I think it’s kind of crazy how much people share on there, so yeah.

Swyx [00:12:59]: ‘Cause like if you’re there, like you’re very, like you’re exactly the right audience?

Alex Zhang [00:13:03]: Yes, exactly.

Swyx [00:13:03]: Like this isn’t gonna reach the mainstream.

Alex Zhang [00:13:04]: And there’s a lot of like introductory material as well, that we’ve put on, that I think is useful for people.

Benchmarks, Princeton, and Research Taste

Swyx [00:13:10]: KernelBench was kind of influential. I just wanna see like, that was last year.

Alex Zhang [00:13:13]: Yep.

Swyx [00:13:14]: What other ongoing work do you wanna shout out that people should pay attention to? ‘Cause obviously you’re involved in this

Alex Zhang [00:13:20]: Yep

Swyx [00:13:20]: Field.

Alex Zhang [00:13:20]: Yeah. I will give maybe the background story of like I am actually involved in a lot of benchmarks, or I used to be, maybe prior to my PhD. It started because I was at Princeton. I worked with the SWE-bench team there.

Swyx [00:13:33]: John, Carlos

Alex Zhang [00:13:33]: John, Carlos

Swyx [00:13:34]: Ofir

Alex Zhang [00:13:34]: And Ofir. They’re all great. Like

Swyx [00:13:36]: Karthik

Alex Zhang [00:13:36]: I love them. Yeah.

Swyx [00:13:37]: There’s basically this, like I think people don’t understand how much benchmarks come from the same group

Alex Zhang [00:13:42]: Yeah

Swyx [00:13:42]: At Princeton.

Alex Zhang [00:13:44]: It is crazy.

Swyx [00:13:45]: Do Xun Yu?

Alex Zhang [00:13:45]: Yes. Yeah.

Swyx [00:13:46]: We had him on a pod before. Now he’s like running Tencent.

Alex Zhang [00:13:48]: Yeah, now he’s like, he’s like a superstar. when I met him, so he was advising my friend Michael Tang, who is now at Anthropic, but they worked together a lot. We were like the two undergrads in Karthik’s lab. I-- And then some others joined later as well. But yeah, Xun Yu is great. I did not know he was like such a superstar until like later on, like after I left, but

Swyx [00:14:12]: Yeah. like, okay, so there are very few PhD students. Like yours is like the next one. Like once a year, we feature someone like who is like basically entire PhD, has been like on target.

Swyx [00:14:25]: There’s not that many of them. Xun Yu was like clearly one of them. And, Jack Morris is another one. And like, we talked, before the show, we talked about research taste.

Swyx [00:14:33]: Right? Like somehow some grad students just have a very blessed career where like, yeah, mostly like, yep, this is like going to stick around, relevant, everyone should know this.

Alex Zhang [00:14:42]: Yeah.

Swyx [00:14:42]: And then others, just nothing.

Alex Zhang [00:14:44]: I think this is also true of like even people within like industry labs as well. I think it’s just like grad students are a lot more visible. So you just see, like you see, like, there are some people who

Swyx [00:14:56]: Yeah, you can publish

Alex Zhang [00:14:57]: Who really like get lucky and like, or it’s, it’s a mix of being lucky and also being very smart and things like that. I think like with research taste as well, like I think it gets developed through opportunities, at least in my case. Like I got-- I was very fortunate to have like taken the path that I took, like working at Princeton and then like later, like finding my like Omar at MIT. Like he’s a fantastic advisor. I will say, though, I find that the most successful research from grad students or like in academia comes when people care about problems that maybe like most people in industry are not looking at. I think this is the issue that like a lot of grad students work on things that benefit, like that look good to an industry lab. Like for example, they’ll work on some, like some benchmark that’s really popular now. I think benchmarks in 2023 were a very different story than benchmarks now. There are a lot of people that work on like harnesses and like meta-harnesses and like specific harnesses for XYZ task. And when you really think about it, the reason someone would work on this is like maybe there’s like a clear goal shaped around the models that we have today of like, this is what I want to see. But like, I’ll give, I’ll give the like RLM, like the recursive language model paper as an example, because I think like it’s a super simple idea. I think when it came out as well, like there were a lot of people that were like, when they see something like that, they’re like, “What is even the purpose of this?”

Swyx [00:16:27]: Or like too cool.

Alex Zhang [00:16:28]: Yeah. Like why, like what? This is just subagents or something, right?

Alex Zhang [00:16:32]: And I think it’s like when you get a reaction like that, it’s almost like a good sign in the sense that like it’s clear that people aren’t thinking about what the purpose of this is. And I will give another example of like SWE-bench. When SWE-bench came out, Ofir loves to tell this story. When it came out, like nobody cared. Like everybody was like, “This is an impossible task. Like why would we ever even consider this as a benchmark?” And it wasn’t until Devin came out that everyone was like, “Whoa, like this is something we wanna hill climb.” And I think this is, this rings true for. You tend to see that a lot of ideas. I think like the. My favorite, I guess, example of this is Eric Seligman’s work, with like STaR and like Quiet-STaR. Like I think when you read the paper, at least when I first read the paper, I was like, “Is this not like an obvious idea?” Or maybe not. I don’t know. I was like, “Oh, this seems really simple.” Or like chain of thought, and the same thing. Or like Xun Yu’s react. It’s like, okay, like, yeah, sure. But then like when you really think about it’s like why. What is the value of the paper? And I think it comes from like, it tells a bit of a story as to like what you want the field to look like. And that is something that it’s very hard to do this in academia because if you look at all these papers, Quiet-STaR, ReAct, RLMs, SWE-bench, none of these papers are. It’s not like a GPT-6 Astro release? It’s not like everyone’s like, “Oh my gosh, like I’m gonna use this now and this is the best thing in the world.” Like academia just can’t afford to do this, at least right now. I. There’s a whole slew of reasons why I think that should change, but I think it’s like. If you don’t have. As a PhD student, I think you’re in such a unique position where you can work on literally whatever you want for the most part. If you’re not taking advantage of that and working on things that, like, nobody cares about or, like, people see as some trivial thing, like, “Oh, I thought about this, but, like, I don’t use it,” I just think, like, in the end, the research is just never gonna be that interesting because you kind of need to take big bets if you’re gonna be in academia. Because otherwise, I think, like, just go to an industry lab. Like, they have tons of resources, tons of talent. Why constrain yourself in an area where you don’t have a lot of resources and, like, there’s not even that many people around? And I think it’s just. it literally just comes down to, like, big bets. like, you just have to take big bets, and, like, a lot of them will fail? Like, that’s just. it’s, it’s natural. But I think

Alex Zhang [00:18:55]: That is, as a PhD student, like, that’s the biggest advantage you have over any single person at another lab because you don’t have to deal with bureaucracy and all these other things.

Swyx [00:19:07]: Fair enough.

Alex Zhang [00:19:07]: Yeah.

Swyx [00:19:08]: I ask a lot of people this question, and usually they hand-wave away. So I think. I appreciate that you’re actually giving a thoughtful response on Like, no, like, this is your unfair advantage because everything else is biased against you, basically.

Jev and Breaking the Autoregressive Decoder Paradigm

Alex Zhang [00:19:20]: Yeah, exactly. And so, like, it’s honestly. I will bring up Jev as an example because it’s, it’s not an academic

Swyx [00:19:26]: Wow, okay.

Alex Zhang [00:19:27]: It’s not an academic project.

Swyx [00:19:28]: Yes.

Alex Zhang [00:19:28]: I want to bring this up because this also happened with RLMs and, it happens with many other works. Like, things get overhyped, right? To an extent, like, something gets overhyped and then people are like, “Why is this overhyped?” Like, “This is trivial. This is stupid.” And I saw the same thing with Jev because I think the release was like. there is this whole thing about, like, academics, or they’re not an academic group, but, like, people have to do branding and they have to, like, kind of market their research. And so, like, I understand, but I think there was a lot of discourse about Jev just being, like, something we’ve known for years. And I think it’s kind of missing the point of, like, why is such a system so interesting? It’s why is it not just some stupid NLP classifier that, like, we’ve, we’ve been doing, back in our intro ML classes or something? I think what’s really interesting about Jev is that it kind of opens up this question of, are language models correct? Like, in the form that they’re in, can we consider a different design space other than text-to-text? Because what they’re doing is they’re basically saying like, “I will take advantage of this language model backbone. Like, I know it captures a lot of information about language, but I’m going to change the output space of the model to give you a trade-off, which is I will do very fast inference over.” Like, if you have some prior about this problem, like, let’s say I only need to make a binary classification. Am I gonna ask my language model to do this and pay, like, a 400X cost? Like, no. That’s-- it’s, like, silly, right? and I think for the longest time, because the labs are the only places that control, you’re never gonna use something other than, like, GPT-4 or GPT-6 or Fable because they’re the best models. But because of that, like, people have gotten kind of accustomed to this idea that a language model is just a autoregressive decoder. Like, we have accepted this. And I think when RLMs came out, it was the same thing. Like, one of the comments, like a very frequent criticism I got was like, “This is not a language model.” Or like, “When I look at this, like, I thought it was a new architecture, but it’s actually not.” And my response to that is like, “Well, a language model is just modeling language. It doesn’t have to be this transformer decoder,”? and Jev is really interesting in that, like, we now have a new

Alex Zhang [00:21:49]: Thing to tune, which is like, what is the output space and how does this affect inference latency? and I think we can actually start asking this about various parts of the language model itself. we are seeing this too with, like, loop transformers. It’s a similar idea of a lot of the attention around it was like, “This is a silly idea.” Like, “Why? Who cares about this?” But it’s like, it is a simple idea, but it’s actually. it opens up a whole new set of questions that I think, like, especially if you’re a PhD student, these are the things that you wanna answer. Because I think it’s like we don’t know. For Jev, for example, we don’t know how far we can take this. for loop transformers, we also don’t know how far we can take this. What if you loop only a subset of the model? what if you route to, like, only. like you have some router to different parts of the model? Like, can you mimic what you do in a harness inside of the model? What can you bridge between a harness choice and a model architecture choice? These are all questions I think that open up with works like this, and that’s, like, really exciting. Jev in particular, when I saw it, I was like, “This is actually really useful for RLMs.” Like, I think it’s, it’s. it makes sense ‘cause the biggest bottleneck in RLMs or swarms or systems like these is they’re slow. When you do multiple language model calls all the time, you’re not distributing your compute correctly because, like, maybe there’s something trivial that you just want a simple model to do, but you can’t do it because your language model is just this bulky thing? So I’m very excited. I think we will start to see new types of models emerge beyond just the bog-standard frontier model, and that is like. there’s so many things that you can do with these, like, new trade-offs.

Swyx [00:23:34]: I’ll also shout out Thinky with their interaction models.

Alex Zhang [00:23:36]: Yes. Yeah. Another great example.

Swyx [00:23:38]: Yeah. So, like, basically try to break the paradigm from sequence to sequence, decoder only, and just literally do anything else.

Alex Zhang [00:23:46]: Yeah.

Vibhu [00:23:46]: I think there’s, there’s a level of if you’re trying to compete, you’re not gonna compete with a Frontier lab doing an autoregressive

Alex Zhang [00:23:54]: No.

Vibhu [00:23:54]: Decoder on. Like, the amount of compute scaling resources they Even Thinky will not. okay, there may be one of the handful that can, but you’re not really gonna do much in that at least.

Alex Zhang [00:24:06]: I don’t know too much about Thinky, or I don’t wanna say anything either, but it’s like if their strategy is just to replicate OpenAI or Anthropic, like that’s a horrible strategy.

Alex Zhang [00:24:15]: Because, well, because, like, they just don’t have. Like, you kinda just have to think of it in terms of, like, what advantage do you have? And if you’re going to use the same setup. I’m sure they’re not, but it’s like if you’re going to do the same setup, like you’re basically competing on the things you can control, which is data and compute, and obviously they cannot compete with the Frontier Labs on that. So yeah, it makes sense that, like, if you are a neo lab, like. Actually, I don’t know if you would consider them to be a neo lab, but I guess, like

Vibhu [00:24:40]: Yeah. That’s why they’re, they’re in there.

Alex Zhang [00:24:42]: I guess they’re kind of a weird one, yeah.

Vibhu [00:24:43]: They’re in their list. They shipped Inkling. Like, they can’t.

Alex Zhang [00:24:46]: Anything other than OpenAI or Anthropic, maybe like Meta and GDM, like you just, you gotta do something else? Like, it just. It’s the sad reality, but I think. I actually think it’s a good thing. I’m very happy that, like, scale works and these companies will just keep doing it because it opens up, like, potentially new players, like if they uncover something really interesting. Because I sort of have my doubts that this is, like, seriously going on at Frontier Labs, ‘cause it’s like why would you do that? like, why would you take the risk of allocating a large amount of compute to new bets when the old bet already works? So

Vibhu [00:25:24]: And I think that’s what spins off a lot of neo labs, right?

Alex Zhang [00:25:27]: Yeah.

Vibhu [00:25:27]: You have a side bet and you don’t get compute, and you’re like

Alex Zhang [00:25:30]: Yeah

Vibhu [00:25:30]: “Okay, I’ll go, I’ll go do that.”

Alex Zhang [00:25:31]: Exactly.

Vibhu [00:25:31]: And, your example of the potential upside is something like Jev, which is X hundred times cheaper, comes out, and maybe it is language model.

Calibration, Fast Classification, and New Model Trade-offs

Alex Zhang [00:25:41]: Yeah.

Vibhu [00:25:41]: In this case, it’s just different.

Alex Zhang [00:25:42]: Yeah.

Swyx [00:25:43]: Yeah.

Swyx [00:25:44]: So no speculation on what Jev actually is?

Alex Zhang [00:25:46]: I guess I have some guesses for what it might be. I have seen some people say like, “Oh, it’s like a diffusion thing.” I guess that

Swyx [00:25:56]: Which is the parallel decode, right?

Alex Zhang [00:25:58]: Yeah, parallel decode. Honestly, I think regardless of what it actually is, ‘cause I think you can. I’ve seen some, like, open-source replications of it. What is really exciting to me about what they did is I’m not entirely sure what their optimization objective was and how they trained it. And I think, like, this is a thing for RLMs that we’ve also been thinking about, which is like, okay, like RLMs are a very simple idea. If I come out with this paper, like anyone can use it now. But what distinguishes The actual value of an RLM is whether or not you can train it properly, and whether or not maybe you can mold some architecture around the system to make it really good. And that’s something that, like, I’m actively working on, I guess. But I think for them, like, they figured out a way to train the system, which is completely non-trivial. Like, I actually don’t really know how they did it. And I’ve seen some comparisons online of, some people are claiming they used Qwen, or they post-trained on top of Qwen, but every open-source Qwen that you use is gonna be worse, ‘cause whatever they did to train it clearly works very well. And so that’s, that’s very exciting.

Swyx [00:27:04]: There’s one element of calibration Which, is a rare topic that I don’t think people even knew about or understood. We covered it with, our conversation with Clementine Foley of Hugging Face, and she used to run the evals, at Hugging Face, which is basically the idea that, models are attuned to give you the most likely next token. But, they’re gonna lie to you when you ask them, “How confident are you?” Because they’re just gonna give you the most likely next answer instead of, like, actually, like, no, let’s calibrate. Like, I am actually fifty percent sure, or I am twenty percent sure, and, like, let’s try to calibrate that. I would say, like, if anything, I think that actually that’s pretty easy to generate synthetic data around Because you can sort of see the truth and then synthetically generate a bunch of answers, have Jev classify it, and then compare with ground truth.

Alex Zhang [00:27:52]: Oh, I see.

Swyx [00:27:52]: That would be my reverse engineering of this.

Alex Zhang [00:27:54]: Yeah.

Swyx [00:27:54]: I’ve actually. I think calibration is probably the under. Like, people are just using it as a very fast classifier But they’re actually not even using the probability or calibration estimates.

Alex Zhang [00:28:04]: Yeah.

Vibhu [00:28:04]: I think it’s also still just misunderstood to reiterate. When you ask a model, “How confident are you?” it will spew out what, forty-three percent. the big delta is this is a grounded classification, right?

Alex Zhang [00:28:16]: Yeah. Yeah, I’m, I’m very excited to see what people do with this model. is it gonna solve everything? Like, no, of course not. But I think it solves a class of problems that we traditionally struggled with, which is low-latency things. So I love the examples with games. That’s actually like. I

Swyx [00:28:34]: Yeah, the Doom example

Alex Zhang [00:28:35]: Yeah

Swyx [00:28:35]: Was very good.

Alex Zhang [00:28:35]: I have a benchmark on language models playing video games. I’ve always been fascinated by whether you can have an intelligent system play new games and things of this nature. And so I think it’s really cool that they have sort of a unique way to do this, to capture language and understanding in, like, a fast, a very fast model.

Language Models Playing Video Games

Vibhu [00:28:59]: Oh, while you’re on the topic, anything you wanna point out for video games?

Alex Zhang [00:29:02]: Oh, yeah.

Vibhu [00:29:03]: This is. you did do a benchmark on any project, right?

Alex Zhang [00:29:05]: So, yeah. I guess these numbers are very outdated

Vibhu [00:29:08]: Yeah.

Alex Zhang [00:29:08]: Because a lot of the models are very different. And I’ve seen actually people run. There are some folks out there that are running newer models on these games, which is really cool. I guess the general premise of this benchmark was we just want to see if vision language models are good enough at just, like, plugging into games with the latency constraint included. ‘Cause this actually. I came out with this right after Claude Plays Pokémon came out.

Alex Zhang [00:29:33]: So this was, like, two years ago, which I guess is, like, ancient now. But I think what’s really cool about this suite of tasks, it’s very diverse in terms of what games they are. And also, I think most of the games are games that people know or, like, have seen before. I saw, yeah, Jeff playing Doom. I will say I don’t think. I think they were just playing, like, really simple levels and stuff. But honestly, like, most models still can’t really do. Or I don’t actually think any models can solve these games very meaningfully. Like, there are some games that they can. I think I’ve seen Astra be able to solve the Kirby game. And we also, for this benchmark, we intentionally designed a really minimal harness. And I will get to this point about harnesses because I think there’s a whole conversation to be had about, like, what is the value of a harness? Like, what is even the purpose of a harness for a model? But in general, like, I think it’s. yeah, I hope to see very quickly or very soon, like, all of these games beaten by newer models.

Vibhu [00:30:35]: Yeah, it’s interesting. Like, the old Cloud Place Pokemon, they, like, read state from RAM and saw what tiles are walkable and whatnot. We did a podcast with them A long time ago.

Alex Zhang [00:30:45]: Gotcha.

Vibhu [00:30:46]: Yeah. Just fun.

Swyx [00:30:46]: Yeah, and it’s similar. Like, Jeff doesn’t have vision

Alex Zhang [00:30:48]: Yes

Swyx [00:30:48]: So you have to kind of feed in,

Harnesses as Compositional Generalizers

Alex Zhang [00:30:50]: Yeah

Swyx [00:30:50]: These, like, game state and all these things. let’s go right into the harness stuff

Alex Zhang [00:30:54]: Awesome

Swyx [00:30:54]: Because you brought it up. Language model harnesses are compositional generalizers.

Alex Zhang [00:30:59]: Yes.

Vibhu [00:31:00]: You struggled to read that one.

Alex Zhang [00:31:01]: Explain. Yes. Okay. So I have been a little unsatisfied maybe with how people think about harnesses, because people compare like, “Oh, like, I love Claude Code, I love Codex, I love Pi.” Like, “No, I love Oh My Pi, I love Prime Agent.” To be honest, I think all of them are the same. Most of the design decisions or, like, the design choices around these harnesses are the same. Maybe Prime Agent is a little bit different because it’s, like, inherently an RLM. But in general, like, I think we can be a lot more creative with harnesses. And what by that is if we think about this from the perspective of what exactly is the harness doing for the model? Well, basically, when you’re trying to solve a problem and you want to use a language model to solve it, like a very difficult task, one thing that we have discovered is that next token prediction is a really awkward form to do a lot of these tasks. So for example, take SWE-bench. When you’re navigating a code base, like, are you going to be able to figure out how to do all of this with a single language model call? Like, you just say, “Solve code,” or like, “Solve my query over this code base.” No. So we rely on a harness to help you do these kinds of things. And I think what is interesting is, like, a harness is a very opinionated program over how you want a language model to be form fit over a problem. And I bring this up because when we think about the, what a harness is doing, we should really think about, like, what choices in the harness let the language model solve this task? And can I actually just have a language model that just does this? Because a harness, if you think about it, now that loop transformers are a thing, I think what’s really interesting about it is you can model a looped transformer in some ways, like, with a harness as well, right? You’re just looping over the model. Now you can say like, “Oh, I’m not decoding,” so it’s, like, a little bit different. But in general, we, for whatever reason, have stuck with the same model architecture choice forever. And I. And there’s many arguments for why, but clearly, like, we are now training models around harness rollouts. And so there is this very awkward way of doing training over harnesses, which is that we train a language model to act within a harness, but, like, now it’s like a really long, maybe, like, multiple agent rollout that we’re doing. And there’s, like, really awkward, hacky ways of doing this. So what this blog talks about is like, well, one way you can think about what is going on here is if the harness is basically helping the model solve a particular task, can different harness design choices actually do something a little bit more meaningful beyond just, “Here are some tool calls that will help you. Here is a way to grep through your code base.” And so this actually. The idea for this blog came with the RLM idea as well. We just didn’t package it that way. And I think this is actually true. There are many other ideas around RLMs that, like, we will be coming out with, but were all there from the beginning. these are all design decisions around. I think with what is. What I like about the RLM is that there were many iterations and versions of different abstractions that I was interested in doing, and ultimately the RLM made the most sense. But there’s a lot of reasons that aren’t public as to why that’s the case. you will see. So in this example, one of the things that we see with an RLM is that if you sufficiently offload context and write. ask the model to write code over that context, you get this really weird but useful property, which is that when the model recognizes during training how to solve a task, it turns out that the solution to many tasks is very similar across, like, tasks where you don’t even. Like, it’s not even that clear to you that the solutions are similar. So in this example, we have, like, a retrieval task and we have, like, an aggregation task, and they’re very different query. Like, the domain is just completely different. And when you train a regular language model over these two tasks, the trajectories look very different. And so what you’re relying when you. Like, let’s say you use Pi or Claude Code or something, which is not in this blog, but we do have these results. You’ll find that, like, these harnesses distinguish too much between these problems, even though the solutions are the same. And so one thing that we find when training RLMs is that, like. When it sees these problems, it’s the same. And the reason it sees these problems as the same is the sub-agent sees different problems, but the sub-agent is solving an easier sub-task, and so you’re confident it’s smart enough to do it. But for the base overall strategy, they end up looking the same. And so when you train on the left task, for example, it can immediately solve the right task. And so if you go down to, like, the plots that we have, one thing you’ll kind of observe is that the. as you just naively train your RLM on these tasks, they naturally learn to generalize, for example, to longer tasks, because the strategy is basically the same. You’re just modifying, like, a length variable. And this actually also holds for tasks that are different, and they’re, it’s not even-- they’re not different across length. They’re completely different tasks, math tasks versus writing tasks. But the solution, the, like, meta high-level solution is the same. And so when you train the RLM on one of them, it generalizes this behavior to the second one. And there’s no magic here. I guess maybe that’s the thing that I wanna kind of stress. Like

Vibhu [00:36:38]: How would you kind of verbalize what they are learning? So I think in here you say you train it on short tasks, they generalize to stuff 8–30x longer.

Alex Zhang [00:36:47]: Yeah.

Vibhu [00:36:48]: They are learning how to solve these type of problems, or what’s the, what’s the core thing they’re actually learning?

Alex Zhang [00:36:52]: Yeah. They’re learning how to solve these types of problems at a certain length. And it turns out that when you take the strategy that they learned, it is directly transferable to the longer length. Like, they’re

Vibhu [00:37:05]: Yeah.