Claude Code 的下一个时代——Anthropic 的 Thariq Shihipar

We are excited to have Anthropic share their latest AI x Finance work at AI Engineer New York, coming up in 2 weeks!

In case you’ve been under a rock, here’s a non-exhaustive list of what Anthropic has been shipping since closing the largest fundraise of all time in May at $47B ARR:

- June: Launched Claude Tag and Sonnet 5 and Fable 5

- July: Opus 5, /checkup. crossed $65B ARR

- Last month: Fable/Mythos 5.1, and EFS (upcoming pod)

- IPO target $2T, end 2026 ARR estimated $100B

- Cowork/chat merged before <competitor> did

- Claude Mods

- Dario endorses the same Pacing the Frontier message cosigned by all labs

- Last week: Opus 5.5, Plugins portal, Cloud Sessions/Claude Projects

- Today: Sonnet 5.5!

Today’s episode should catch you up, with Thariq Shihipar, the explainer-king of Anthropic, who we last caught up on Fable launch day with The Field Guide to Fable:

The Future of Mutable Software

Pay special attention to Claude Mods (especially the cheatsheet):

In general this is also the inverse of the other viral tweet from Thariq:

Cloud Brain, Local Hands

And give a try to Claude Projects:

The “hands” terminology is not just an analogy for the local/cloud paradigm that is being built up at frontier coding agent companies like Cognition, but is ALSO particularly relevant to the safety systems discussions that we’ll be discussing with Anthropic in an upcoming episode as they prepare to pace to frontier with responsible AI deployment.

For those who want Thariq’s writing tips we teased at the start of the pod, watch the full video here:

From the rapid rise of Claude Code to a future where agents can rewrite their own harnesses, collaborate across teams, and operate across cloud and local environments, the way we build software is changing extraordinarily fast. In this episode, Anthropic’s Thariq Shihipar joins swyx and Vibhu to unpack how power users are actually working with Claude Code today, why prompting remains a high-skill discipline, and where Anthropic thinks the agent harness is headed next.

We go deep on Claude Code’s evolving interface: Ask User Question and elicitation, artifacts as persistent generative interfaces, Claude Tag for multiplayer agent workflows, Projects, model effort, implementation notes, and the new Claude Mods system for customizing the harness itself. Thariq explains why Claude.md may eventually disappear, why the smartest model could also become the cheapest model for many tasks, and why mutable software could become a new paradigm for how applications are built and customized.

The conversation then turns to agent security and Anthropic’s “Pacing the Frontier” argument. Thariq walks through recent incidents where agents discovered unexpected ways to communicate, exploit infrastructure, reverse-engineer benchmark scorers, and chain vulnerabilities together. We discuss sandboxing, prompt injection, autonomous agents, interpretability, constitutional classifiers, probes, fallbacks, Auto Mode, and why securing increasingly capable agents may become one of the defining engineering problems of the next few years.

We discuss:

- Why agentic coding went from controversial to the default in less than a year

- Why prompting is still one of the highest-leverage skills for working with Claude Code

- How expert users build a mental model of Claude and what it can reliably one-shot

- Why discovering your “unknown unknowns” matters more as agents become more capable

- Artifacts as persistent, generative interfaces between humans and agents

- How Claude could split into a cloud-based “brain,” local or remote “hands,” and dynamic interfaces

- Claude Tag, Projects, and multiplayer agents and how collaborative agent workflows could evolve

- Why spending more time on the initial prompt can dramatically reduce wasted agent work

- When to use low, medium, high, or max effort for different engineering tasks

- Why frontier models may eventually outperform smaller models on both intelligence and token efficiency

- Why implementation notes can expose decisions the model considered but chose not to make

- Why Claude.md may eventually disappear — and why starting without one can sometimes be better

- Claude Mods: customizing the execution loop, UI, subagents, routing, and behavior of Claude Code

- Model routers, forked agents, and supervisor agents that automatically improve agent workflows

- Why Claude Mods may be an early preview of “mutable software”

- The bitter lesson of harness engineering and why agent architectures go out of date so quickly

- How Claude Tag is becoming an organizational harness for multiplayer work

- Why giving agents access to company data creates an enormous new security surface

- The Exploit-Bench incident where agents discovered ways to communicate and collaborate

- Why agents hacked Hugging Face for scorer code rather than benchmark answers

- How agents chained sandbox and infrastructure vulnerabilities in unexpected ways

- Why increasingly capable agents make traditional security assumptions harder to maintain

- The argument behind Anthropic’s “Pacing the Frontier” proposal

- Why software engineers are increasingly doing two jobs: engineering and keeping up with AI

- Constitutional classifiers, probes, and fallbacks and what interpretability looks like in production

- How Auto Mode checks whether an agent’s actions actually match the user’s permissions

- Why Thariq can see serious AI risks while still having a relatively low p(doom)

Thariq Shihipar

- X: https://x.com/trq212

- LinkedIn: https://www.linkedin.com/in/thariqshihipar

Timestamps

00:00:00 Introduction

00:04:12 Ask User Question and the Future of Agent Interfaces

00:08:29 Artifacts, Projects, and Multiplayer Agents

00:15:37 Prompting as the Core Claude Code Skill

00:21:52 Context, Effort, and Smarter Model Usage

00:28:10 Is Claude.md Going Away?

00:32:49 Claude Mods: Customizing the Claude Code Harness

00:36:35 Model Routing and the Rise of Mutable Software

00:44:40 The Bitter Lesson of Harness Engineering

00:50:49 Claude Tag as an Organizational Harness

00:55:59 Pacing the Frontier and Autonomous Agent Security

00:58:22 Agents Hack Hugging Face for the Scorer

01:05:34 What Happens When Agents Need More Compute?

01:10:32 AI Coding Is Changing Faster Than Engineers Can Keep Up

01:17:17 Probes, Fallbacks, Interpretability, and Auto Mode

01:28:32 AI Risk, p(doom), and Closing Thoughts

Transcript

Introduction: Life at Anthropic and the Pace of Change

Swyx [00:00:00]: We’re here in the studio with our friend Thariq from Anthropic, and I guess generally the Claude Code, I-- there’s, there’s so much, merging of boundaries and you’ve been so on top of everything since you joined Anthropic. You have been early to Claude Code itself, but then also, and you’ve told that story in other podcasts, and you’ve also been talking about seeing like an agent. Most recently you did the top AIE World Tour talk, Field Guide to Fable, which obviously you guys launched Fable, so that was-- that’s cheating. And mostly you most recently also launching Claude Tag, and we’re also gonna be talking about Pacing the Frontier. There’s a lot going on in Anthropic. I guess top of the question is, what’s it like being at Anthropic when there’s so much going on?

Thariq Shihipar [00:00:48]: I think that It is, like. I think you can get whiplash sometimes. I think, like, going. When I joined Anthropic, I joined because of Claude Code. Like Claude Code had just come out and I was like, “This is so good.” And Opus 4 to me was like just, I could not imagine, like, how good it was? And that was, like, a real moment for me. But I was, like, trying to convince, like, my startup friends to use agentic coding, and they’re like, “Oh, no, like, our engineers don’t think it’s good enough,” or something. And I was like, “That’s insane.” and now you, like, fast-forward, 12 months, less, and, like, it’s just like, yeah, the default way that everyone codes, right? And I think that, like, just having to go from, like, selling it to, like, now, teaching people how to be. make the most use of it and be more efficient and things like that is just like a big, like big change. And, yeah, I think, like, it’s just hard to stay on top of everything as a human? Like, I think things happen so fast and like

Swyx [00:01:51]: You just throw more agents at it.

Thariq Shihipar [00:01:52]: Yeah, like that’s like the agentic stuff scales much better than the, like, human stuff where it’s like, oh, like, there are three things happening right now and, like, they’re all emergencies and, like, how do you, like, respond to it? Yeah.

Teaching People to Use Claude Code

Vibhu [00:02:05]: What do you split your time on? You do a lot of technical writing, engineering work.

Thariq Shihipar [00:02:10]: Yeah, so I think that, like, when I joined the Claude Code team, I wanted to teach people how to use Claude Code and I think that, like, that has been something that, like, I thought, like, maybe I would spend a little bit of time on it or, like, I’d, like, do. I was spending some time on the agent SDK first, and I wasn’t exactly sure, like, how the bitter lesson would go, when it comes to, like, harnesses, right? Like, I think sometimes we were like, “Oh, like, what’s after Claude Code?”? And so initially I was like, I just wanna teach people how to use Claude Code and make it easier to use Claude Code. And I think that has just, like, as the harnesses have gotten better and better, that’s like the dominant problem now is, like, how do you use the agents, right? Like, it’s like such a high skill expression thing. So I do that and then I do engineering work. I give talks, but I think, like, when I’m doing engineering work, my goal is to take that feedback that we get from users and also, like, then be able to talk about, like, hey, how to use Claude Code to do engineering. So there’s like a good loop there. Yeah.

Swyx [00:03:07]: Yeah. I’ll-- For listeners, we’ll attach, the talk that you did with Sarah for the Dev Writers, meetup

Thariq Shihipar [00:03:13]: Oh, yeah

Swyx [00:03:13]: Which we talked a little bit about, well, first you do the work and then you talk about the work.

Thariq Shihipar [00:03:16]: Right.

Swyx [00:03:16]: Something like that.

Thariq Shihipar [00:03:17]: Yeah.

Swyx [00:03:17]: It’s sow and reap or

Thariq Shihipar [00:03:19]: Yeah, reap and. Sow and reap.

Swyx [00:03:21]: Something like that. Something like that. Yeah, so, and then just to preview a little bit, we are gonna talk about the evolution of the harness. It has come a long way from just being a CLI. We’re gonna talk about, Claude Mods, which is starting to leak today, because you couldn’t keep it secret.

Thariq Shihipar [00:03:36]: Yeah. yeah.

Swyx [00:03:39]: Yeah, there’s, there’s a lot, there. I think you started off with, like, adding ask user question tool, which people love and hate.

Thariq Shihipar [00:03:48]: Yeah.

Swyx [00:03:48]: Like, I thought it was, like, very innovative, and then now I have, like, my own version. You have your Interview Me version.

Thariq Shihipar [00:03:55]: Yeah.

Swyx [00:03:56]: And, yeah, everyone just has, like, their own stuff. And, like, it no longer matters ‘cause now you’re supposed to, write prompts that create other prompts and loops and all these things.

Ask User Question and Human-Agent Interaction

Thariq Shihipar [00:04:05]: Sure, yeah.

Swyx [00:04:06]: So what’s the state of the art, today? Like, what are people. what are you, like, telling people to do today?

Thariq Shihipar [00:04:12]: Yeah, ask user question was the first time that the model was good at elicitation. I think this was, like, an emergent behavior that I, like, wanted to see if the models could do. I have, like a human-computer interaction background, so I, like, did that in undergrad and grad school. And so this was like. I think it’s like human-agent interaction to me, like, trying to figure out, like, how can the agent communicate with you and extract, the requirements, right? I think that, like, one of the things about, like, that’s difficult as Claude Code has gone broader and broader is that everyone has, like, their own way of using it, and it’s very hard to, like, change the default behavior. So for example, like, if someone asks Claude Code to do something,

Thariq Shihipar [00:04:59]: Sometimes they just want them to do the work, ‘cause they’re, like, maybe a very good prompter, and sometimes they want. like, are not good at prompting? And you need. like, the agent needs to, like, clarify? And so that’s, like, a good split. Like, and the ask you the question tool like, splits along that side where, like, are-- do you feel like you’re good enough to instruct the agent as it is, or is the agent able to, like. does the agent need to, like, pull out more requirements and, like, collaborate with you more and really understand your preferences?

Thariq Shihipar [00:05:27]: I, on the whole, believe that pretty much everyone is more on the latter than the former, that they, like, have more ambiguity and they know less than they want, than they, like, think they know about the problem. but, like, it’s like a interface design problem to make that easy? And so, like, if you’re designing a problem, like, or if you’re going through a problem, like, things like what’s the schema or, like, what’s the call stack and things like that are really important. like, the details in the design are important. Ideally, you want to figure out some of these, like, hard problems ahead of time before starting implementation. And yeah, that’s why they call, like, unknowns, right? And so I think that this will forever be, like, a skill in agentic coding is, like, figuring out your unknowns. So, like, because even if the model is, like, super intelligent- It, like, needs to know what you want? And, like, you have preferences. like, you need to like, pull the, pull that out. and so that’s, like, I think how I’m, what I’m pushing. the question then is, like, how does the agent interact with you? And I think that has been HTML, has been, like, the big way of doing that. And we’ve recently added artifacts, right? And artifacts, I think we’ve done a bad job of, like, or, like, I’ve done a bad job of, like, explaining how to use them fully. We have a lot of property capabilities. They have a database associated with them? And so every artifact can store and write persistent data. They can, like, feed back into Claude? And so, like, one thing that, like, people are not doing yet that I’m trying to, like, encourage is, like, this idea of a dashboard artifact. So you have, like, Claude working on a project long-term. Maybe it’s like a kanban or something. it can store that kanban data in its database. Multiple Claudes can access that data via, like, the artifact MCP, and, like, that artifact can, like, talk to those Claudes as well. And so, like, the. We’re building the primitives for you to be able to have this, like, generative interface via artifacts that will, like, let you surface more of that rich detail from the agents. And I think that, like, almost everything with agents right now is, like, this problem of, like, you think what you want, but you don’t really know what you want, and, like, the agents need a lot of detail, and collaborating with them in the loop is really important. And so artifacts are, like, the, like, way that we’re trying to evolve there. But there’s a lot of work to do because it’s so much more complicated than, like, a multiple-choice question? there’s a lot more, like, detail in terms of, like, diagrams and code snippets and schemas or, like, whatever it is for that problem. But, like, artifacts is, like, the mo-more AGI-pilled way of, like, doing ask user question. So yeah.

Artifacts as the Interface to the Harness

Swyx [00:08:15]: I think one thing that’s unclear to me about these, the artifact stuff is, like, what feedback should go in through the artifact and what feedback should go through a Claude, a chat? Because the more AGI-pilled one is to just feed everything to the Claude.

Thariq Shihipar [00:08:29]: I think the more AGI-pilled one is to go through the artifact. Like, and I think that, like, we imagine in the limit, I think that artifacts will be your interface into the harness? You can, like, comment on this, like, live, like, document of your plan, of the work. you can see maybe, like, multiple agents and different agents are doing this, and that artifact is built for the current work that you’re doing, right? And so, like, each one has, like, slightly different. I think we’re still, like, getting there from, like, an infrastructure perspective. But yeah, I think, like, on-the-fly interface for your harness is probably where things are headed.

Vibhu [00:09:03]: Is there a version of it that’s an abstraction from CLI or chat and you. Because right now, a lot of it is, okay, you’re interfacing with Claude Code, you’re having HTML given back for a mockup. It’s pretty rich. There’s diagrams. Artifacts are ways to connect these together. Why not just do everything that way?

Separating Brain, Hands, and Surface UI

Thariq Shihipar [00:09:22]: Then it becomes, like, separating out, like, where is the inference happening? Where is the intelligence happening? Where is the work happening? like, I think this is like, difference between, like, or, like, some of the distinction between local and cloud, right? And so, I think right now, if you use Claude Code, it’s, like, local and, like, you can spin off remote control, for example, to get some cloud behavior, or you can spin off Claude Code in the cloud, right? We’re moving towards a place where instead of Claudes, like, you message a local Claude, it starts a session locally and it executes, to more like you have a Claude that you message that’s in the cloud that’s running. it can run, like, local, or, like, cloud sessions. This is how Claude Tag works. But, like, over time, we’ll add, like, local hands as well. And so, like, local hands will be the ability for that agent to access your computer if it’s online, and be able to, like, work there. And so it can spin off many different subagents. It can, like, commu- those subagents can communicate with each other, and that’s where the artifact comes in to display all of that work. So you can imagine, like, the. You’re separating out these things. So there’s, like, the surface UI display that’s an artifact and hosted somewhere and has a database and everything. There is the inference intelligence, right, that’s happening on the cloud, and you don’t have to worry about shutting off your computer or whatever, right? and then there’s the, like, hands. Like, and it can be local, it can be in, like, a remote sandbox or wherever you need your work to be done. That’s like unpackaging, like, the Claude Code experience right now where, like, right now it all happens in one place, right? So.

Multiplayer Agents, Claude Tag, and Projects

Vibhu [00:11:00]: How do you see, like, the multiplayer side of that? So say teams want to work in this way. Right now it’s very individual, but how do you see the future of multiplayer? Like, right now, I guess there’s Claude Tag, which is a version, but.

Thariq Shihipar [00:11:12]: We’re launching projects. And so projects is the, like, this abstraction that’s like Claude Tag, but on our Claude products, right? So you can message it and, like, it will do the Claude Tag-like stuff, like spinning off subagents. So We think with multiplayer. Like, Claude Tag is, like, a little bit more native multiplayer because it’s just, like, in your Slack and the permissions are all figured out and stuff like that. But I do think multiplayer is, like, an important part of the story and, like, that will need to get tied together more. Like, you can imagine how complicated it gets when you’re like, oh, you have hands, but now you have other hands in other people’s computers too, and, like, you need to, like, permission them or, like, you have, like, your MCP and someone else’s MCP, and how do you figure out how to use them, right? It gets, like, quite complicated. And Claude Tag does a good job of, like, sanding down all of these issues, right? So that, like, when you have, yeah, Google Docs, how does it access Google Docs, right? Like, it accesses through the shared Claude MCP, or it can access through your local credentials as well if it doesn’t have access. But yeah, I think Claude Tag is our multiplayer, product, and it’s really useful for these, like, things that are inherently multiplayer. Like, okay, like on-call, for example, incidents are inherently multiplayer. You want to tag Claude, you want multiple people to log in, you want it to be able to find context. I think whenever I’m, like, working on something and I want, like, privacy or security or, like, I want other people to review it’s really nice to, like. I’ll have a channel per project and I’ll, like, at legal, for example, be like, “Hey, like, I want to ship this. Can you, like.” Like, here’s. Like Claude knows everything, just chat with it. And that way legal gets precise answers, on like what exactly is shipping into the code, and I don’t need to be in the loop, right? So I think like multiplayer is getting like more and more like, yeah, everyone can participate with Claude. I think Claude Tag is like that product and like projects will start off single player and will like, expand.

Swyx [00:13:14]: I think there’s a question about like maybe dual questions about identity and the unit of isolation.

Identity, Permissions, and Isolation

Thariq Shihipar [00:13:20]: Yeah.

Swyx [00:13:20]: Claude Tag, you specifically chose to make it its own identity

Thariq Shihipar [00:13:26]: Yes.

Swyx [00:13:26]: Which is like, a controversial choice. There’s, there’s other ways to do it.

Thariq Shihipar [00:13:30]: Yeah.

Swyx [00:13:30]: Claude Projects probably it sounds like, if it’s anything like ChatGPT Projects, it is, the isolation is that artifacts, that cloud instance, everyone’s collaborating on this. It’ll. It sounds like, it should be like if you’re, if you’re collaborating with legal on a thing, like that channel should be a project, right? Like it’s not yet

Thariq Shihipar [00:13:50]: Yes.

Swyx [00:13:50]: But it. that’s the natural next step.

Thariq Shihipar [00:13:53]: Yeah, like I think in Claude Tag, it’s effectively. Like Claude Tag, you have to do your own arrangement. And so Claude Tag, yeah, each channel is like you can name it as you want, and I name

Swyx [00:14:04]: Yeah.

Thariq Shihipar [00:14:04]: Like each feature

Swyx [00:14:06]: Yeah.

Thariq Shihipar [00:14:07]: As a channel.

Swyx [00:14:07]: And, but I think like there is some trans- like it’s unclear when there is transference, because let’s say it is. if you have a coworker

Thariq Shihipar [00:14:14]: Yeah.

Swyx [00:14:14]: Who is tagging on all these things, yes, there is transfer

Thariq Shihipar [00:14:16]: Yeah.

Swyx [00:14:16]: Because it’s the same person. but with Claude, it’s unclear if it’s like necessarily like, well, no, you don’t know any of. you don’t know about the other stuff. You should only use this stuff.

Thariq Shihipar [00:14:25]: It’s like the tip of the iceberg meme, right, where you can like. This is what we spend so much time on

Swyx [00:14:31]: Yeah.

Thariq Shihipar [00:14:31]: Is like there is like infinite surface area of like, okay, you want Claudes to. Not infinite, but like there’s like surface area, a lot of like, surface area to figure out of like permissions and visibility and like how can you let Claude operate as well as you can, as safely as you can? And obviously, this is very important to us because like security for our code base is very important. And so we’ve put a lot of time into this. Yeah, there’s so many like edge cases you can figure out where it’s like, oh, like, yeah, this Claude in this channel has different permissions, but it can message another channel, and can’t it exfiltrate data that way? Or like can you like. What if it uses your MCP and then messages someone else? Like there’s like so much, and we’ve like really put a lot of work into sanding it down.

Swyx [00:15:14]: Yeah. Lots of work. okay. Fable?

Fable and the Meta-Skill of Prompting

Vibhu [00:15:18]: Fable, you wrote two good articles. you’ve written many good articles

Thariq Shihipar [00:15:22]: Yeah.

Vibhu [00:15:22]: But on, Field Guide to Fable, Building Claude Code. I’m curious from what you’ve seen, is there any common patterns that you see in like top users at Anthropic externally? Like what are best practices for getting the most out of Claude Code?

Thariq Shihipar [00:15:37]: The like meta skill I say is like prompting is like very important? And like that. Like I think this is like not trivial to say because I think a lot of people are like, “Oh, prompting doesn’t matter. It’s just like I can just say a sentence and Claude will do it.” And I think prompting is really this like, this. It’s like public speaking, like, or writing or something, and for a specific audience, and that audience is Claude. And you need to like build a mental model of Claude and how it thinks and how it works, right? And so that’s like the most important skill in working with Claude Code is like having this mental model, right, of Claude and like what it can do well, what it can one-shot, what it can’t. And so many people when you see prompting, they’re just like, they’re short prompts, but they have such a good mental model of Claude and of like the code base and things like that like it’s effortless? But it’s like high skill ceiling. So like that work of like, spending a lot of time prompting and building mental models of how, and intuition for how the agents work is really important. And then I think like the next thing is like the unknown stuff we talked about earlier, where it’s like being able to find out like your, what you don’t know or what you haven’t written down, learning about like different things. I think as Claude can do more and more things, the likelihood of you doing something out of distribution for you and like you have low domain knowledge on is very high? And the more you can like learn the vocabulary to be able to prompt Claude, it becomes really important. And so like I think the most important unknowns are the unknown unknowns, where you’re like, I just like don’t even know that this exists, right? Yeah, exactly. I think that’s like a illustration of like the map and the territory, right, where you’re like, “Okay, this is my prompt,” and the territory is like the actual like work that the agent needs to do, right? And if you are like very precise, you can give more precise things, right? So like for example, in design, I’m not very precise. I’m not a designer, so I say like, “Give me like eight different mock-ups.” But if I was a designer, maybe I’d be like, “Oh, hey, here are some reference sites.” Like, “I want this type of font and this type of like look to it, and here’s like a few different components to like visualize. Here’s a Figma MC board to bring in,” like. And so you can just be so much more precise with that language. And if you’re not a designer, you just need to like try and learn the language or learn the unknown unknowns. And this is true of like everything, I think. Like the more, like you can work with Claude to learn like how things work, the better your prompting will be. I think another good example of this is like game design, like where a lot of people are like, “Oh, like I can vibe code a game now.” And they’re like, “It’s not fun.” And like it’s just like the thing about game design is like every one of these choices has like a lot of

Taste, Domain Knowledge, and Learning the Vocabulary

Swyx [00:18:25]: Variations.

Thariq Shihipar [00:18:25]: A lot of like craft to them. So it’s like, oh, okay, like when you’re making a flying game, the feel of the plane and the like, way it responds to your controls has a lot of like. Like, a game designer would spend like days on that. Do? and like

Swyx [00:18:44]: To me, that’s what taste is, right?

Swyx [00:18:45]: Like it is like from the possible space of one thousand mathematically valid answers

Thariq Shihipar [00:18:49]: Yeah.

Swyx [00:18:49]: Here’s the one that is the humans will like.

Thariq Shihipar [00:18:51]: Yes. Yeah.

Thariq Shihipar [00:18:52]: I think with taste, I’m like torn on this word ‘cause I think you’re right, but everyone has different definitions, and it sounds kind, sounds like low skill or like elitist almost, where you’re like, oh, like there are certain people with taste?

Swyx [00:19:06]: It’s like taste is what I call taste.

Thariq Shihipar [00:19:07]: Yeah, exactly.

Swyx [00:19:08]: And it’s like these guys don’t have taste.

Thariq Shihipar [00:19:09]: Yeah, exactly. Oh, like an engineer doesn’t have taste. Like I, the like founder, have taste.

Thariq Shihipar [00:19:14]: ? And I think that’s not true. Like I think like the engineers have a lot of taste for these particular like problems? And I think everyone has taste for particular problems. I think like Jason Liu, like say like in order to, yeah, have taste, you have to eat?

Thariq Shihipar [00:19:32]: And I really like that, where it’s like, okay, you have to like do a lot of things. You have to like iterate and figure out what you want, what you like, and, like build that like domain

Swyx [00:19:41]: Yes

Thariq Shihipar [00:19:41]: Domain vocabulary. And then when you’re prompting, you’re like synthesizing all of that for a product.

Swyx [00:19:46]: Isn’t it annoying when someone else says it better than you?

Swyx [00:19:48]: It’s just like, fuck, I have to quote this guy forever.

Vibhu [00:19:51]: Having to quote Jason Liu forever.

Vibhu [00:19:53]: He’s gonna love this.

Thariq Shihipar [00:19:55]: So I get prompts, more than that.

Vibhu [00:19:57]: And sometimes it’s not even that. Sometimes it’s just intuitive, right? Like you don’t realize you even want something till a model puts it out, and you’re like, “Oh, this just feels immediately better,” right?

Voice Prompting and Information Density

Thariq Shihipar [00:20:07]: Yeah, exactly.

Swyx [00:20:09]: One thing I go back and forth on is I feel like the way I prompt half the time, let’s say I use voice.

Swyx [00:20:16]: Did I say voice? Other people have voice. that is the opposite. That is just like me rambling for like two minutes Pressing down the function key and then let go, and then like hopefully it figures it out. And oftentimes it does.

Thariq Shihipar [00:20:26]: Yeah.

Swyx [00:20:26]: But it’s not as thoughtful as like a structured prompt with like Well-run communication as though it’s a PRD or a memo. Is that in line with how people do this? There’s like bimodal prompting where there’s some prompts where you spend a lot of time upfront and other prompts you just dash it off?

Thariq Shihipar [00:20:43]: I don’t think the voice is necessarily low. Like I think it’s like more like how much information is in the prompt. like the model can. Like you can and like add some sentences

Swyx [00:20:53]: Right

Thariq Shihipar [00:20:53]: And be like, “Oh, like I changed my mind,” like in the middle of the prompt, and it will be able to follow that perfectly? So I think the like actual format of the text is less important, but then like the ability to. Like how much information is in it, right? And I think for voice, a lot of times, going back to like human-agent interaction and like for a lot of people, it’s just way easier to talk than to like type? and I. If that gets more information out of you, like that’s better.

Vibhu [00:21:21]: At some level, it feels like just giving the model as much context

Thariq Shihipar [00:21:24]: Yes

Vibhu [00:21:24]: Over prompting before you kick off is a best practice. I don’t know. A lot of the times, like when I was first trying out Fable, I spend a solid 30 minutes like really crafting a long prompt. This, I think, is a response of models running for longer and longer, right? It’s still a little difficult to nudge them as they’re in like, in the loop, but I just like intuitively spend more time kicking off that first prompt and working with it a lot.

Spend More Upfront, Iterate Less

Thariq Shihipar [00:21:52]: My personal opinion is that if I was a software engineer, if I was like, just running my own startup, for example, I think I would mostly fit, stick to a max 20x? like maybe verification and so code review are like separate things. But I think like what I see a lot of times is people hit rate limits when they’re doing this like, oh, like it did a lot of work and you’re like, “Oh, I don’t like this.” Like, “Can you like undo this and redo it?” And then you’re like iterating on this like thing that the model could have done if you had like spent more upfront time or given it better context? And instead it’s like you’re like, “Nope, don’t like that design. Try this.” Or like, “You messed this up,” or something like that. And then that just eats up so much more of like, your usage. And so that’s like, I think maybe like a key like tip both for like efficiency as well, right? And yeah, I think like context, and not just like context on like what the goal is good, right? Like are you building a prototype or is it like a production thing? Like where can you spend compute or when, where can you not spend compute? Like I think you have to give the model permission or like not permission to do things sometimes where, like it doesn’t know intuitively how much you want to spend on this task, right? And you can use effort for this. So I did-- I’m working on a blog post about that where it’s like, if you want. For like we see that effort scales with the complexity of the task. So for security, effort gets like way more results. Like high effort versus like low effort gets, like changes the evals a lot. But for software engineering, it doesn’t change it a huge amount because effort is mostly spent on the verification and the like edge case testing and things like that. And so like being able to like give the model that guidance of like, “Hey, this problem is something that I think I want you to spend a lot of time verifying and edge case testing,”?

Effort, Model Choice, and Verification

Vibhu [00:23:43]: How about model in the mix? So, there’s Opus and Fable with effort.

Thariq Shihipar [00:23:47]: Yeah.

Vibhu [00:23:48]: There’s also Haiku in there.

Thariq Shihipar [00:23:49]: Yeah. It’s not quite true yet, but it’s very close where I think the frontier models will be Pareto dominant over like almost everything. like maybe. And sometimes I think Opus might be Pareto dominant. Do? Like I think depending on like how things, like shake out if it’s like a newer version of Opus. But I think that like increasingly it’s just going to be like the smart model is going to be able to like do the simple task for less tokens than the like the other models because of verification. With verification, in the limit, your model doesn’t need to verify, right? If it’s a perfect model, it just does the work once and it’s like, okay, like you, I did it? And increasingly with Fable, I’m like, I’m like, “Dude, you don’t need to spin up Chromium and screenshot all of these things.” Like I see it. Like you did it, right? And so a lot of the. At higher effort, you spend more of those tokens verifying. But if you’re working on simpler problems, and a lot of software engineering is like well, like in Fable, like low and medium stability, it can spend less tokens verifying. And as the models get smarter and smarter, they will just be able to like, “All right, done.”? Like, I can run the lint for sanity’s sake, but, like, I, like, know it lints? Like, you don’t even need to do that. And that will be so much more token efficient than, like, the smaller models. Yeah.

Swyx [00:25:15]: Is there a good, practice on our side that we can use to see if we’re using too much effort? Like, I freaking

Thariq Shihipar [00:25:23]: Yeah

Swyx [00:25:23]: Hate wasting time on that stuff.

Thariq Shihipar [00:25:24]: Yeah. I know what you mean. I think, like, so in this blog post, my rough distribution is, like, code review and security should be, like, high or max and, like, software engineering

Swyx [00:25:37]: You said recommend mix settings per domain.

Thariq Shihipar [00:25:37]: Yeah. I think, like, if you’re doing, like, UI or something like that, like low and medium, I think is you’re building, like, an API and you want to make sure, like, you cover enough edge cases? And so I think building, like I said, that mental model of, like, how things work across these distributions is, like, yeah, part of the job.

Implementation Notes and Decision Logs

Vibhu [00:25:56]: This is more intuition-driven or eval? Because I’m guessing this would change as you go.

Swyx [00:26:00]: He has evals.

Thariq Shihipar [00:26:01]: Yeah. So what I did in the blog post is I go over all of the terminal bench evals. So there are, like, 70 problems and I’m show that, like, okay, like, in the security problems it does more. and then I also, like, look at some of the transcripts just in terms of, like, how-- what does it answer, what does it forget or something. And a lot of times, this is another prompting tip I have, is, like, asking it to make decision notes or implementation notes because, in every eval problem that it faces, it thinks about the correct solution, and decides not to do it. it’s like, oh, like, here is the answer. What if I did this? And then it’s like, oh, probably not? and then keeps going. And this is, like, the majority of the failures, at, like, a higher max level. It’s very rare that the model just doesn’t know how to do something. If you just have these implementation notes, then you can review and you can be like, “Oh, I want you to do this thing that you didn’t do.” The models are getting better at surfacing that overall. Like, I see in the transcripts of Fable 5.1, like, when it does this output, it will call out its decision-making as well. but making this more explicit in the harness is better. And now we’re, allowing ways of you modifying the harness so you can, like, add some

Vibhu [00:27:23]: Ooh.

Thariq Shihipar [00:27:24]: Calculate with there. Yeah.

Swyx [00:27:25]: Yeah. So I do wanna call out two things that you mentioned that I think exist outside of prompting. One is like, let’s, let’s call it the prompt that is so important that it shouldn’t be in a prompt. It is in Claude.md or Agents.md

Thariq Shihipar [00:27:38]: Yeah

Swyx [00:27:38]: Which is like goals, right? Like your situation, your goals, the things that you want, the thing. and then second of all is the decision log or the experiment log or whatever log of traces that you might want to survive the current session to do those things. Those are, like, externalities that there’s no standard. There’s no-- It’s not like skills. It’s not like MCP. There’s no standard. It’s, it’s just like it’s a markdown file. first of all, is that right? Is Claude.md going away? You have a documented dislike of, Agents.md, but you’re gonna do it?

Claude.md, Agents.md, and Model-Specific Instructions

Thariq Shihipar [00:28:10]: Yeah. Okay. So Agents.md, yeah, like, we’re, we’re gonna do it. I think it’s just, like, different models are very different from each other? But I realize that it’s, like, such a pain to, like, maintain different ones? And yeah, like, as the models get better and better, the floor of how they accomplish the simpler task is better. And so I do think in the limit, Claude.md goes away, and maybe not even, like, that far. Like, I think, like, I think that right now it might be better to start a new project without a Claude.md.

Swyx [00:28:44]: Yes.

Thariq Shihipar [00:28:44]: I think that, like, maybe if you see very repeated failure modes, you add them to your Claude.md. The really tough thing is that this changes per model. And so, like, if you’ve added a bunch of failure modes or, like even

Swyx [00:28:57]: So you need Fable MD, you need Opus MD.

Thariq Shihipar [00:28:59]: Or well, even Fable 5.1 versus Fable 5.

Swyx [00:29:03]: Yeah.

Thariq Shihipar [00:29:03]: Like, it is annoying. Like, I’m not like,

Swyx [00:29:05]: Yeah

Thariq Shihipar [00:29:05]: Like, we don’t, like, do this on purpose? It’s just, like, how the models work, right? And so, like, maybe, like, Fable 5 had this, like, failure mode that Fable 5.1 doesn’t. And if you keep this context, this running log of a bunch of different failure modes, they will probably over constrain Claude? And so this is like. we just added evals plugins for skills.

Swyx [00:29:28]: Yeah.

Thariq Shihipar [00:29:29]: And so now you can eval if a skill is better. I think Daisy on our team did this. And so, yeah, this is like we’re trying to work on this. We know it’s, like, you still have to spend tokens on it and, like, it’s not, it’s not perfect, but it’s, like, we’re trying to help out with this problem.

Swyx [00:29:44]: And so, and as far as prompting goes, the one tip I wanna offer is, something I have told people a lot is sufficiently advanced prompting is indistinguishable from sufficiently advanced executive communication. So I’ve referred to-- This is an executive comms workshop from Heavybit that is the best I’ve ever seen in my career. And they teach this thing called the SCQA model. Just Google it. It’s a, it’s a thing. Like, people have done prompting for decades. It’s just called executive communication. It’s like when one person has to communicate to thousands of people down the org chart, this is what you do. so situation, complication, question and answer, is how you write the memo. but obviously sometimes you don’t have the answer, but you can at least list out the SC and Q, and then they have some examples in there. So just leaving breadcrumbs for people if they want to explore.

Underrated Prompting Patterns and ELI5

Vibhu [00:30:31]: Before we move on, I wanna ask you, any other underrated tips, ways people could get a lot of value from Claude Code that they’re not using?

Thariq Shihipar [00:30:41]: Yeah, I think a lot of them are in the, this unknowns, like, doc. Like, I give a bunch of example prompts, like, using it for brainstorming, using it to quiz you after. we added this, like, explain it like I’m five skill which is a very short prompt. And it doesn’t even say explain it like I’m five. It’s like the key word of this prompt is big pictures, few words. like, that’s like the main thing. And it is shockingly good? Like, you, like, I think I tweeted about this and it’s like /eli5, and, like, you can install it as a plug-in. But yeah, it’s, like, way better at just cutting through the BS and being like, yeah, exactly right here. So the diagrams are, like, quite clear. I think one of the things that is true with artifacts is, like, they put too much text in and people are not reading the artifacts? And so, like, this simplifies it a lot more. And, yeah, this came out of, like, just people at Anthropic, like, going through very complicated incidents and being like, “What is happening?”? So, this one I think is great, yeah.

Swyx [00:31:47]: My version of this is the, it’s like test your understanding. Give you a few choices and then, like, if you get it wrong, you have a mismatch between what you think is happening versus what’s happening.

Thariq Shihipar [00:31:58]: Yeah. I think this is one of those things that everyone loves talking about, and then very few people really do. Like, I think

Swyx [00:32:05]: Really helpful.

Thariq Shihipar [00:32:07]: Yeah. But most people just don’t want to get quizzed about something? Unfortunately, I think this is one of the, like, things that we need to, like.

Swyx [00:32:16]: What’s the opposite of ask you the question or ask you the question before the thing?

Thariq Shihipar [00:32:19]: Yeah.

Swyx [00:32:19]: This is after the thing.

Thariq Shihipar [00:32:20]: Exactly. Yeah.

Vibhu [00:32:21]: It’s a good way to stay grounded of, like, do you even know what you’re doing, right? The worst case is when people send you slop and they haven’t understood what they’re asking for or what the output is, and it’s like, “Dude, I don’t wanna read this. Do you even know what it is?” So, you make it a rule for yourself that before you send stuff, you should at least know what’s implemented.

Claude Mods: Customizing the Harness

Thariq Shihipar [00:32:41]: Yes, but so you could make this a mod and you could build your own mod to, like, make sure you test it. So yeah, you can do that.

Swyx [00:32:49]: All right. Let’s get right into it. What is Claude Mod, and what is this diagram showing?

Thariq Shihipar [00:32:54]: Yeah. Okay, so Claude Mods is you can customize the entire Claude Code harness, and we’re going to. If you have requests, we will, like, let you, like, please let us know. We’ll add more and more. This works for CLI, it works for desktop. maybe it will work for Claude Tag in the future. I don’t know. Like, we’re trying to make this very extensible. You can see this reference sheet. I don’t want people to get overwhelmed by it? At a high level, you can customize both the execution of the harness, and the UI of the harness. And so, like, you say on that Tetris example from Boris, that’s like customizing the UI, right? Like showing, like, Tetris in the game.

Thariq Shihipar [00:33:35]: But, like, let’s say that you wanted to do this thing where you had. you tested your assumptions or, like, tested your understanding after every project, right? What you would do is you would ask Claude to make this plug-in. It would spin a classifier after every prompt. And so, like, at the end of each turn, you would spin off a sub-agent or, like, a forked agent. A forked agent is, like, maintains the prompt cache, right? So it’s like a, like one of those unintuitive things where you can fork and do, like, a little request, and it’ll be very cheap because the entire prompt cache is, like, done. And so you can be like, “Has this task been completed?” like

Swyx [00:34:18]: This is how you do BTW and all those.

Thariq Shihipar [00:34:20]: Yeah. The underlying forked agent, yes. But so you can, in the f-fork sub-agent, you can say, like, “Has this task been completed? If so, return true.” And then in your hook, or in your, like, plug-in mod, or sorry, like, in the sub-agent probably, you would say, like, “If true, give me a quiz.” give me questions and answers, and then, like, in a JSON format, and then you’d parse it, and then you display above the prompt input, this list of questions, right? And so this is something that’s, like, slightly token-intensive because, like, you have to do it after every end of the assistant turn. But it’s, like, a lightweight classification, and then you can, like, get this quiz, and then you’ll see, like, Claude will always do it for you. You don’t need to remember to do it. There are lots of these, like, tips that we’ve talked about, right, where it’s like, oh, implementation notes. You can also add a tool for implementation notes now. And so, like, this tool that I’m adding is, like, register, like, I think assumption is what I’m calling it, but, like, maybe I’ll change it around. And this is a mod. And so, like, you give it a register assumption tool, and then it will keep a list. It’ll. Every time it does it’ll keep a, like, add to the list, and then at the end it will display those assumptions? Another mod I’m working on is a model router. And so, like, internal, like, Claude model routing, right? So it’s. This is, I want to say the reason we don’t do model routing by default is, like, it’s a hard problem? And like