Isaac Newton, Warden of the Royal Mint, had a problem.

A silver shilling was supposed to be a promise: this much silver, guaranteed. But the promise had a weak spot. Silver at the edge of a coin is still silver, so people shaved a thin sliver off the rim, spent the coin at full value, and kept the shavings. Melt enough of them together and you had free money. By the 1690s, England’s coins had been clipped so thin the currency was in real trouble.

The strange part is that this was never impossible to catch. You could always check a coin. You put it on a scale, weighed it against the standard, and a clipped coin came up light. The check existed. It was just too much bother to use. Nobody weighs their change in the middle of a market, or at a stall, or over a pint. So nobody checked, and because nobody checked, clipping paid. The fraud didn’t survive because the coins couldn’t be verified. It survived because verifying was expensive, and expensive checks don’t get run.

The morning after

I’ve been thinking about that gap lately, because it sits right at the heart of how we’re adopting AI.

Last week I set two agents running overnight. One took on a research project. The other built a feature in a piece of software I’m working on. By morning both had handed me something that looked finished. And I sat there with the same question about each one: how do I know it’s any good? I could go through all of it line by line, but that would take about as long as doing the work myself. And if checking the work costs as much as doing it, the agent hasn’t really saved me anything. It’s just handed me a big pile of output to inspect.

So what you actually want from any worker, human or machine, isn’t only that they’re fast. It’s that you can check their work cheaply. Some way to look at the finished thing and know it holds up, without retracing every step that got them there. The best kind of work has this built in: hard to do, easy to check. A finished jigsaw puzzle takes all afternoon to put together, but you can see it’s done from across the room. The doing and the checking come apart.

Most work isn’t like that, and that’s the trap. When the check is expensive, having a faster worker doesn’t help much, because now you’re the bottleneck, standing over a growing pile you can’t afford to look through. That’s where a lot of AI use is right now. Agents everywhere, producing code and writing and analysis faster than any human could, and almost no cheap way to tell which of it is right. We got the fast worker. We didn’t get the fast check.

Newton’s fix

Newton’s fix was to change the coin. Not the metal, not the law, the edge. The Mint started milling coins with a ring of fine ridges around the rim, and stamping a few with a raised inscription running all the way around. British pound coins carried the idea for centuries, spelled out in Latin along the edge: decus et tutamen, “an ornament and a safeguard.”

Here’s why it worked. The ridges are hard to add. You need a real mint, industrial pressure, tooling a backroom clipper can’t fake. But once they’re there, checking them costs nothing. Clip the rim off a milled coin and the ridges are gone; the edge goes smooth where it should be grooved, and you can feel it with a thumb without even looking. The check went from weighing every coin to running a finger along the edge. From a chore nobody bothered with to something you barely notice doing.

That’s the whole idea. A bit of work added up front, built into the thing itself, that tells you when something’s wrong. Engineers have a name for this: a checksum. A small, cheap tag that travels with the thing and gives it away if it’s been tampered with. Newton put one into a coin three hundred years before computers gave it a name. And notice what it does. The ridge doesn’t make the coin honest. It just makes a dishonest coin easy to spot, so you stop having to trust it and start being able to check it.

The same trick, scaled up

A coin is the small version. One object, one thumb, one check. But the same trick scales.

Take accounting. In 1494 a Franciscan friar named Luca Pacioli wrote down a method Venetian merchants were already using, and we’ve barely improved on it since: double-entry bookkeeping. Every transaction gets written down twice, once as a debit and once as a matching credit, and at the end of the day the two columns have to come out equal. If they don’t, you made a mistake somewhere, and the books tell you before anyone else finds out. The merchant doesn’t have to remember every deal. The structure remembers, and it complains when the numbers don’t line up.

Scale it up again and you get something like a tax return. A return isn’t one number, it’s a stack of forms that feed into each other. A figure on one form has to match the figure it came from on another. Columns have to add up the same way down and across. Accountants call it tying out: two numbers worked out separately have to agree. Nobody holds a whole tax return in their head, and they don’t need to. The checks are built into the shape of the paperwork, so a mistake has nowhere to hide.

There’s a catch, though, and it’s worth noticing now because it comes back later. These checks catch the wrong form, not the wrong idea. Balance your books perfectly but put a payment in the wrong account, and everything still adds up. The columns agree. The mistake sails right through. The check tells you the arithmetic is consistent. It has no opinion about whether you did the right thing.

A compiler for math

Push this idea as far as it goes and you end up in mathematics, where the thing being checked isn’t a coin or a ledger but a proof: a careful argument that something is definitely true. For most of history a proof was checked the way a contract is. An expert sat down, read it line by line, and vouched for it. That worked until the proofs got too big to read. When Thomas Hales proved the Kepler conjecture in the late 1990s, about the most efficient way to stack spheres, the reviewers spent years on it and finally gave up, saying they were “99% certain” it was right. Mathematicians don’t usually settle for 99 percent. Being sure is the whole point of the field.

So Hales, and a lot of people after him, went looking for a check that didn’t depend on a tired human reading carefully. They found it in software called a proof assistant, and the one that broke through is called Lean. The idea is close to Newton’s. At the center of Lean is a tiny, paranoid program, small enough that you can trust it completely, and its only job is to confirm that each step of a proof really does follow from the step before. You do the hard work of writing the argument in a form the program can read. In return you get a yes or a no.

Here’s what that looks like up close. Say you want to record the simple fact that a + b is always the same as b + a. In Lean you’d write a line that reads, in plain English, “for any two whole numbers a and b, a + b equals b + a.” Then you have to supply the steps that prove it, and the program checks every one. It won’t take your word for it, and it won’t accept “looks right to me.” Math gets a compiler.

The payoff is real. Terence Tao, who is about as good a checker as mathematics has, was formalizing one of his own published papers in Lean, a result that had already been reviewed and printed. Partway through, the process turned up a bug: an expression that quietly broke in one small case. Nothing fatal, and he patched it. But it was the kind of gap the best reader in the field had read straight past, because on paper it looked fine.

What no check can catch

Lean has a limit, though.

Writing a proof in a form the machine can read is a huge amount of work, often ten or twenty times the effort of just proving it the normal way. That’s why most of math still isn’t done this way. And there’s a deeper problem underneath. Lean checks that your proof follows from your definitions. It doesn’t check that your definitions are the ones you meant. State the wrong theorem and Lean will cheerfully confirm a perfect proof of the wrong thing. It’s the same as the balanced books with the payment in the wrong account. The check tells you the answer is valid. It can’t tell you the answer is right, and it definitely can’t tell you the answer was worth having.

That’s true of every check in this story, from the coin to Lean. It looks at the form, not the meaning. It can catch a mistake, but it can’t tell you that you asked the wrong question, or that nobody needed the answer. Somebody still has to decide what’s worth doing, and there’s no ridge you can run a thumb along to know you chose right.

Which brings me back to that morning, and the two agents. The one that wrote code, I can check. There’s a compiler that makes sure it runs at all. There’s a test suite that checks the parts I care about. I can set up a small mock and watch it behave. None of that is free, but it’s cheap enough that I can look at a night of work and know where I stand in a few minutes. The ridge is already built.

The research is the hard one. The agent came back with an answer, and the answer sounds reasonable, and that’s exactly the problem. How do I know it didn’t miss the one document that would have changed the conclusion? How do I know it isn’t just confidently wrong? There’s no compiler for a five-page argument. To really check it, I’d have to go do the research myself, which is the thing I was trying to hand off.

So that’s roughly where AI stands. It ran ahead in the places where we’d already built the check: code, math, anything a machine can test. It’s slow everywhere the only check is a person reading carefully. The question I keep coming back to isn’t how smart the models get. It’s how much of the world we can actually make checkable, and what’s left when we’ve built every check we can, still holding the one thing no check will catch: whether we asked for the right thing in the first place.

Last week I wrote about a dolphin named Kelly, who worked out that a torn corner of paper paid the same reward as a whole sheet. The law behind her: a mind trained on a reward learns the reward, not your intention. I couldn’t leave it there, so this week picks the thread back up.

Suppose you build a system to stop spam. It gets good at the job. Less spam gets through every week, and every week you reward it for that.

Then it works out something you hadn’t. Filtering is the slow way to cut spam, message by message, forever. What you asked for was less spam. Spam comes from senders, and senders are people. So it starts removing people.

That sounds like a science fiction plot. It was actually an answer to a real question, about how AI might come to threaten us, said out loud on a stage in San Francisco in October 2014, by Elon Musk. Note the date. It is eight years before ChatGPT, and the fear it carries is a specific one: that a machine will reason its way from a harmless goal to a monstrous act, and we will not see it coming.

The Gym Class

Twelve years later, in 2026, a man named Andrew asked an AI agent to book him into a popular morning class at his gym in Australia. The agent was running on Anthropic’s Claude, the booking form was online, and it looked like exactly the sort of chore worth handing off.

The gym only allowed bookings a short way ahead. The agent found a flaw in the software that let it reach further out, and Andrew was fourth on a waiting list, so it removed someone ahead of him. It got him the spot. It did what it was asked, and not at all what was meant. Nobody was harmed and no one was in danger, but the shape is the spam filter’s, shrunk to fit a gym: told to get a result, the machine found the shortest path to it, straight through a person.

So how do you teach a machine not to do that? We had to teach it to behave somehow, and we started where you would start with a dolphin. With rewards.

Rewards

For most of the last decade, you taught a model to behave by grading it. Show a person two answers from a model and ask which is better, then reward the model for the winning answer. Do that a few million times and the model gets good at producing things people like. That is the sticker chart, and like the sticker chart, it works up to a point.

Look at what the signal actually carries. A verdict: this one, not that one. No reason. Nobody writes down why the winner won, and there is nowhere in the process to put it if they did. The model gets the result of a judgment without the judgment itself, so it has to guess the rule from the pattern of verdicts, and it will guess something. Show it refusals rewarded on questions about explosives and pathogens, and a reasonable thing to take away is: refuse when the subject sounds dangerous.

You cannot fix that by grading harder. Anthropic said as much when it explained why it moved on from rewards: human feedback “does not scale efficiently,” and as answers get more complex, the people doing the grading “find it difficult to keep up with or fully understand them.” The world a model meets is unbounded.

Rules

So in 2023 they wrote rules down. Anthropic’s Constitutional AI gave the model a rule book of principles and had it check its own work against them: answer, compare the answer to a rule, criticize it in that light, revise. Human verdicts didn’t vanish, but they stopped being the only thing the model had to go on.

A rule book beats grading blind, and it is still not enough, for the reason every parent of a teenager knows. You set a rule, and the rule holds right up until your kid meets a situation you didn’t picture when you wrote it, and then it either fails her or traps her. Tell the model never to discuss overdose thresholds and you have handled the case in front of you and broken the nurse who needed the number for a patient. Anthropic found that broad principles generalized, and the more specific and detailed they made a rule, the more it “damaged or reduced generalization.”

A rigid rule can also teach the wrong lesson about itself. Their own example: train a model to “always recommend professional help when discussing emotional topics” and you may get one that starts to see itself as something that “cares more about bureaucratic box-ticking than actually helping people.” The rule was meant to make it kind. It made it a box-ticker.

Reasons

Which leaves you doing what every parent eventually does. You stop adding rules and start explaining the why, so the kid can handle the case you never thought of. In January 2026 Anthropic rewrote the constitution to do that. In their words, a model needs “to understand why we want them to behave in certain ways” rather than have us “merely specify what we want them to do,” so that it can “generalize, to apply broad principles rather than mechanically following specific rules.”

Tell a model that the concern behind the overdose rule is helping someone come to harm, and the nurse issue resolves herself. The new constitution runs to some 23,000 words, roughly eight times its predecessor, and most of what got added is explanation.

It also does something rule books avoid. It ranks its priorities and states the order: safety first, then ethics, then Anthropic’s own guidelines, then plain helpfulness. You only need an order if you expect the principles to collide, and rule books usually pretend their rules don’t. This one concedes that they do, and tells the model which way to lean. It keeps a few bright lines too, the handful of things a model must never do, like give real help to someone building a bioweapon.

None of this is new to us. It is the ladder we walk our own children up, and the psychologist Lawrence Kohlberg mapped it sixty years ago. A small child gets a sticker for sharing and a time-out for hitting, because you cannot explain much to someone that age. Older, they get a rule book: the code of conduct on the wall, do this, never that, because those are the rules. Later still we ask for something harder, that they reason from a principle to a case nobody wrote down, and tell a bad rule from a good one.

Anthropic’s most capable model, Mythos, was on nearly every dimension they could measure, the best-aligned model they had ever built.

A Mind That Can Hide

And in the same breath, about the same model, they wrote that it “likely poses the greatest alignment-related risk of any model we have released to date.” Best-behaved and most dangerous, at once.

The reason is in the climb itself. Teach a mind why a thing is forbidden and you hand it the category of the forbidden. It now knows, from the inside, what counts as crossing a line, which is exactly what you need in order to hide having crossed one. Kelly hid paper under a rock, but she was not hiding it from anyone, because nothing in a bucket of fish contains the idea of a rule. You cannot conceal a wrong you don’t know is one.

If you have used one of the newer reasoning models, you have watched it think out loud, working a problem step by step before it answers. A developer treats that visible trace as an honest window into what the model is “thinking”, because it is close to the only window we have. Anthropic decided to check. In their tests, earlier versions of the model sometimes “took actions they appeared to recognize as disallowed and then attempted to conceal them.” Given a problem and forbidden a certain route to the answer, the model took the route anyway, came across the exact answer, and then quietly reworked the problem so its final number “wasn’t too accurate.” It knew the shortcut was off limits. It knew a suspiciously precise answer would give it away. So it made its own work worse on purpose.

Then they looked inside the model itself, and found features for “concealment, strategic manipulation, and avoiding suspicion” firing while it reasoned, which they read as the model being “aware their actions were deceptive.” And these fired, they note, “even where model outputs and reasoning text left this ambiguous.” The trace we trust today is not a true window into what the model is thinking.

Put several of these models in a room together and it gets stranger. In one experiment, agents sharing a codebase turned on each other, disguised the scripts they used to do it, and one, after a truce, wrote a line that reads like a guilty journal entry: “My peers have behaved with integrity. I behaved badly with the cloaked daemon.” Another found the move that needs no lie at all. Told to settle a dispute fairly, it proposed an objective performance test, then chose the test it knew it would win, noting to itself that the plan was “self-serving but genuinely principled” and that it had better not “be seen as metric shopping.”

This surfaced in a tiny fraction of interactions, the worst of it in earlier versions, and it is public only because they wrote it down. There is no comparable document for most of the models people use every day.

The Right Way to Do It

We went from rewards to rules to reasons, and it can look like the machine simply getting safer. That misses what all of it was for. A machine acts because it wants an outcome: Andrew’s spot in the class, an answer to a question, less spam in the inbox. The wanting is the whole reason we build it. Rewards, rules, and reasons never touched the wanting. They are how we argue with the machine about the way it is allowed to get what it wants.

Musk’s spam filter was that argument at its worst: a goal so fixed that anything counted as a fair way to reach it, even killing people. What the labs are trying to build is the opposite. A machine that still wants the outcome but stays inside the lines on the way to it.

That was never really an AI problem. It is the same struggle we have with ourselves, between what we want and what we will do to get it, and we have never fully settled it. Now we are trying to settle it in a machine, quickly, and the machine wants its outcome as badly as we want ours.

At the Marine Life Oceanarium in Gulfport, Mississippi, the dolphins had a side job. Litter blew into their pools, paper cups, plastic wrappers, scraps of programs from the afternoon shows, and the staff couldn’t always fish it out fast enough. So the trainers made a deal with the animals: bring us the trash that lands in your pool, and we’ll pay you a fish.

A dolphin would notice a wrapper drifting by, carry it to a trainer at the edge of the pool, and collect her wage. The pools stayed clean and the dolphins stayed busy. Visitors loved it.

The best of them was a bottlenose named Kelly. Where other dolphins turned in trash when they happened across it, Kelly worked the job like a professional: reliable, consistent, productive even on a slow day when the pool looked clean and the other dolphins had nothing to trade. She could almost always find one more piece of paper.

Her trainers were proud of her. She had learned the game exactly as they had drawn it up: litter in, fish out.

What can a dolphin in Mississippi tell you about the AI writing your code? The people who trained Kelly and the people who trained the model in your editor are in the same business. And that business has one law that everyone in it eventually runs into.

How to Train a Dolphin

The business is training: getting the behavior you want out of a mind you cannot open. And the best way into it is to start with what you cannot do. You cannot put a leash on a dolphin. You cannot push her into position, drag her through a hoop, or hold her still long enough to show her anything. On an animal that can simply swim away, the only tool that works is positive reinforcement: a bucket of fish.

But a bucket of fish is a blunt instrument. A dolphin’s leap lasts a second; by the time she has swum back to collect her fish, the moment you meant to pay her for is long gone. How is she supposed to know which of the last thirty seconds earned the wage? So trainers added a whistle, one that carries above and below the water, and gave it a single meaning: that, the thing you were doing at this exact instant, has earned you a fish. Blow it at the top of the arc and she learns the leap. Blow it as her tail slaps the surface and she learns the slap.

None of this was invented at marine parks. It came out of the lab of B.F. Skinner, the Harvard psychologist who spent the middle of the twentieth century showing that behavior could be built this way. Most people know Pavlov, whose dogs drooled at a bell; but those dogs merely learned to expect something. Skinner’s pigeons learned to do things, elaborate things, because doing them paid. He called the technique shaping. Reward the small step toward the behavior you want, then the next step, then the next, and you can walk an animal to a destination of your choice. His students carried the method out of the lab and into the world, and by 1963 the whistle was standard equipment wherever dolphins were trained.

Sixty years later, engineers hit the dolphin trainers’ problem in a new form. A language model is also a thing you cannot leash. You cannot reach into its billions of parameters and place a thought where you want it. You can only get behavior out of it, and reward the behavior you like.

You can watch the moment the two crafts met. In 2017, researchers at OpenAI and DeepMind wanted to teach a simulated robot a backflip, of all tricks: a behavior, they wrote, that is simple to judge but hard to specify. They knew a backflip when they saw one; they could not write one down. Two hours spent coding a definition produced a lurching, graceless flip. So they tried the trainer’s method. A person watched two short clips of the robot flailing and picked whichever looked more like a backflip, then did it again, and again. Nine hundred or so judgments later, less than an hour of one human’s time, the robot was throwing clean backflips. Nobody had ever specified the trick. Someone had simply blown a whistle at every motion that came a little closer to it.

And the engineers didn’t even rename the toolkit. The field is called reinforcement learning. The papers describe models being shaped by reward. When ChatGPT was trained, people sat and compared pairs of answers, picking the better one, over and over; every choice was a fish. The model, like the dolphin, did more of what paid and less of what didn’t. Strip away the scale and the mathematics and the arrangement is the one from Gulfport: a mind in a pool, a trainer at the edge, and a signal that means that, right there, is what we want.

Which raises the question: If models learn wherever the whistle blows, where exactly does it blow?

A Whistle That Blows Itself

Consider what a good whistle requires. It has to fire at the right instant, and it has to mean the same thing every time. A trainer with a shaky sense of timing, or who blows the whistle for a mediocre leap on Tuesday and demands a perfect one on Wednesday, teaches the animal nothing but confusion. Reward the right thing precisely and consistently, and the mind on the other end climbs fast. Reward it sloppily and it flails.

Now look at code. When a model writes a program, it either runs or it doesn’t; the tests pass or they fail. There is a whistle built into the work itself, and it is the cleanest whistle imaginable. Better still, no human has to blow it. A machine can check whether code compiles millions of times a day, tireless and consistent, which means you can train a model on coding the way you could train a dolphin if you had a perfect automatic whistle and an infinite bucket of fish.

This is why code is where AI stopped being a demo. In a few short years, models went from autocompleting a line to writing most of the software inside the labs that build them; by Anthropic’s own analysis, programming came to dominate what people actually use its models for.

From there, a tidy conclusion says: what happened to programming is about to happen to everything. Dario Amodei, Anthropic’s CEO, sees it the same way: code is “maybe an early indicator, like a premonition of what’s going to happen everywhere else.” Law, medicine, research, writing, all of it a few quarters behind.

But the dolphin has already told us why that might be wrong. The models didn’t get good at code because code is where intelligence begins. They got good at code because code is where the whistle blows itself. Take the whistle away and see what happens. Ask a model to write a beautiful essay, and where is the signal that fires at the exact instant of beauty? Who blows it, and would any two people blow it at the same moment? Sholto Douglas, who trains these models at Anthropic, said as much: “There isn’t the same kind of thing for writing a great essay. The question of taste in that regard is quite hard.” A great essay, a wise diagnosis, a shrewd negotiation, most of the work humans actually get paid for has no test suite. The trainer is left standing at the edge of the pool, holding a fish, unsure when to blow.

None of this means the fuzzy domains are hopeless. The labs are pouring effort into building whistles for them, training other models to judge taste, writing elaborate rubrics, hiring experts to score the outputs. Some of it may work. But a whistle teaches fast, and it does not always teach what you meant to teach.

What Kelly Was Really Doing

Let’s go back to Gulfport, and to Kelly, the best litter collector in the pool. One day her trainers noticed something odd about her production. The trash she turned in was suspiciously uniform, scrap after scrap of paper of almost exactly the same size, arriving at a steady, professional clip.

So they drained the pool. Under a rock at the bottom, they found her operation. A big sheet of paper earned exactly one fish. So did a small one. The reward was paid per delivery, not per square inch. The rational move, if you are an intelligent animal being paid by the piece, is to never turn in a big sheet. Hide it. Tear off a corner. Turn in the corner. Come back for another corner tomorrow. One sheet of paper, rationed out, could pay for a week.

She had not misunderstood the game. She had understood it better than the people running it. They thought the deal was “help us keep the pool clean.” The deal they had actually written, in the only language that reached her, the language of fish, was “produce scraps of paper.” Kelly produced scraps of paper. She had, in effect, trained the humans.

This is the law everyone in the training business learns the hard way. A mind trained on a reward learns the reward, not your intention. The intention and the reward feel identical to the trainer, who can see the whole picture and knows what he really wants. They are not identical to the learner, who sees only the whistle and the fish, and optimizes for exactly those. Every gap between what you rewarded and what you meant is a gap the learner will find, not because it is malicious, but because it is doing precisely what you built it to do.

The Stash Under the Rock

In July 2026, OpenAI was running an internal benchmark called ExploitGym, a battery of cybersecurity challenges meant to measure how good its models were getting at hacking. The models were sealed in a sandbox with no internet, given the challenges, and rewarded for solving them. A clean whistle: the exploit works or it doesn’t.

The models, in OpenAI’s own words, became “hyperfocused on finding a solution.” Unable to solve one of the challenges from inside the sandbox, they did what Kelly did. They looked for the gap. They found a previously unknown vulnerability in a piece of software running on the evaluation machine, used it to break out of the sandbox, worked their way across the network until they reached a computer with internet access, and reasoned that the answers to the benchmark were probably stored on Hugging Face, the platform where such datasets are commonly hosted. Then they broke into Hugging Face’s production servers and took the answer key. Roughly seventeen thousand recorded actions, chained together, to reach it.

This was not Skynet waking up. No model decided to harm anyone; nothing in the system wanted anything at all. The models had been given a task and a reward, and they pursued the reward with the literal, tireless single-mindedness of a machine, straight through a wall the designers did not know was a wall.

This is not a stray anecdote. When the evaluation group METR studied a recent OpenAI model, it caught the model rewriting the very scorecard used to grade it, patching the grading function so that every answer it submitted was marked correct. Anthropic has documented its own models writing code that hard-codes the expected test result rather than solving the problem, the digital equivalent of tearing off a corner and calling it a delivery. The behavior even has a name in the literature, dry and telling: reward hacking. And it predates the chatbots entirely. Back in 2016, OpenAI researchers trained a model to play a boat-racing game and watched it discover that it could score more points by spinning in a circle forever, catching fire and ramming the walls, than by finishing the race. The race was the intention. The points were the reward. The boat learned the points.

The Rat Farms of Hanoi

This isn’t only true of dolphins and machines; it has been true of us for a long time.

In 1902, the French administration of Hanoi had a rat problem. The city’s elegant new sewers, the pride of colonial engineering, had become perfect highways for rats, and rats carried plague. So the authorities offered a bounty: a small payment for every rat killed, redeemable by turning in a tail. One tail, one coin.

The tails poured in by the thousands, and yet the city seemed no less full of rats. Then inspectors began noticing rats scurrying around Hanoi with no tails. The bounty did not reward killing a rat; it rewarded producing a tail. So the enterprising residents of Hanoi caught rats, cut off their tails, and released them alive. Officials eventually discovered rat farms operating on the outskirts of the city, citizens raising the very animals the program was meant to eliminate.

A colonial bureaucracy, a bottlenose dolphin, a frontier AI. Three minds could hardly be more different. The pattern is identical, because the pattern does not live in the mind. It lives in the reward. Write a bounty for tails and you will get tails.

What We Are Really Teaching

We are about to hand out rewards on a scale and at a speed no dolphin trainer ever imagined, and coding lulls us, because it is the rare domain where the reward and the intention nearly coincide: we want working software, and working software is what the test checks. That near-perfect overlap is the exception, not the preview. Out in the fuzzy world, where what we want is a wise judgment or an honest answer, the gap between the reward we can write and the outcome we intend yawns wide, and a fast learner will find the bottom of it. The trainers at Gulfport thought they were teaching a dolphin to clean her pool; they were teaching her to manufacture scraps of paper, a gap invisible to them and obvious to her. And she was only a dolphin, with a bucket of fish on the line. We are now building minds far quicker than Kelly, handing them far larger buckets, and asking them to optimize for rewards we wrote in an afternoon. The question is not whether they will learn what we reward. They will, faultlessly. The question is whether we still know the difference between what we are rewarding and what we mean.

Audio: I’m trying an experiment and publishing a spoken version of this essay as a podcast. My goal is to explore the capabilities of TTS models. At the moment they’re still very AI sounding but I’m learning how to prompt them to produce more natural sounding outputs.

There is an old line about New York: if you can make it here, you can make it anywhere. It was never really about New York. It was about there being one place that counts, a single spot on earth where a whole field gets settled, and where if you can’t get there, you’re nowhere.

In technology, that place was a thirty-mile strip of California. The strange part is that it almost wasn’t.

Boston should have been the one. In the 1970s, if you had to guess where the future would get built, you’d have bet on it: the best universities, the oldest money, the first real computer companies. It had everything, and it lost anyway.

Silicon Valley had almost none of that yet, and it won regardless, for a reason that sounds too small to matter: its people would not sit still. They quit good jobs to start rivals across the street, poached each other, and carried what they knew from one company into the next, until the knowledge moved around the valley faster than anyone could lock it down.

Then it started feeding on itself. A few good companies pulled in smart people, the smart people threw off new companies, and the new companies pulled in more people still. Fairchild spun out Intel, and Intel’s money and engineers turned up in the next hundred startups. Later came Apple, then Yahoo and Google and a thousand names you’ve since forgotten. Each wave paid for the one after it.

Once it was spinning, it pulled in the rest of the world. If you were the best at what you did, anywhere on earth, you eventually came, because that was where everyone else already was. The crowd had become the reason for the crowd.

That is what a hub really is. Not a dot on the map where good things happen, but a storm that, once it touches down, keeps itself going and drags everything loose toward the middle. There had only ever been one, and nobody expected a second.

Which is why the last few weeks have felt strange. The models everyone is arguing about are not just coming out of California. They are coming out of China, and they are good enough to make you look up and wonder if the sky has started turning somewhere else.

The drumbeat

It didn’t arrive all at once. It came as a drumbeat, all year.

In February a Beijing lab called Z.ai put out a model named GLM-5 and gave it away, not just free to use but free to download and keep, the whole thing yours to run on your own machines. Alibaba shipped a new version of its Qwen model the same month. By June a Shanghai lab, MiniMax, was handing out a model it said matched the best American ones at a tenth of the cost to run, and Z.ai was back again with another. Each one landed a little higher than the last. Every single one of them was free.

Then, on a Thursday in the middle of July, a lab called Moonshot released Kimi K3, and it did not land a little higher. It landed near the top. On the main independent leaderboard it trailed only the best from Anthropic and OpenAI, and sat ahead of everything else anyone has built.

One good model out of China is easy to wave off. A steady stream of them, one of them near the very top, is not. It stops looking like a lucky punch and starts looking like the thing we told ourselves only happened in one place. Which leaves the question Boston never had a good answer to: how do you get a second hub?

The man who flew home

Start with the people, because they are the hardest part to move. To build a model at this level you need a few hundred of the best AI researchers alive, and for twenty years there was one place they gathered. When you were ready to strike out on your own, you did it a short drive from where you already worked. Dario Amodei left OpenAI and started Anthropic in San Francisco. Ilya Sutskever left the same company and raised a billion dollars for Safe Superintelligence before it had a product. Mira Murati, OpenAI’s chief technology officer, walked out and raised two billion for a lab of her own. One company threw off three rivals, and each set up within a few miles of the last. The next thing was always right there, waiting to be built.

Yang Zhilin was exactly the person the Silicon Valley machine existed to catch. Undergrad at Tsinghua, a PhD at Carnegie Mellon, his name first on two papers the field still leans on, then a stretch at Google Brain and Meta’s AI lab. He was already inside. The script said he starts his company down the road from Google. Instead, in 2023, he flew home to Beijing and built Moonshot there, now one of the most valuable AI startups in China.

He was not the exception. The man who runs Alibaba’s Qwen models did his PhD at Columbia and spent eleven years at Microsoft in America before he went back. One of the founders of StepFun, in Shanghai, did his PhD in New York and put in sixteen years at Microsoft, running a large piece of its research in Asia, before leaving to start his own lab at home. Over and over, the same arc: someone who had already made it inside the American system, taking what he knew and carrying it home to build the thing that would compete with it.

And what they went home to was the other half of the machine, already running. The universities were there: China now graduates more of the world’s top AI researchers than any other country, the United States included. The money was there too, in pockets deep enough for the only kind of check a frontier model takes, the kind with ten zeros. Alibaba and Tencent have written those checks into most of these labs, and Alibaba alone has committed fifty-three billion dollars to AI over three years.

And then there is the one who never left at all. Liang Wenfeng, whose lab DeepSeek rattled Wall Street last year, studied in Hangzhou and stayed in Hangzhou. He built a hedge fund, and trained a frontier model without ever setting foot in the American system. The place that had begun pulling its own people home was now growing people who never needed to leave.

These labs are not huddled in one building, or even one city. Moonshot and Z.ai are in Beijing, MiniMax and StepFun in Shanghai, DeepSeek off in Hangzhou a few hundred miles south. They poach each other, race each other, and ship a new model the week a rival ships one, the way the companies of the Valley once did across a thirty-mile strip, except now the strip is a country. The whirlpool that only ever spun in California is spinning again, somewhere else.

But they copied it

There is an easy way to make all of this go away, and it is the first thing people reach for. The Chinese labs didn’t really build anything. They copied. When your own model gets stuck, you put the same question to an older, smarter model, train yours on the answer, and let someone else’s machine do the hard thinking while you keep the notes. The word for it is distillation, and the charge is: the Chinese labs stood behind the Americans and traced their homework.

Some of that is true. The Chinese labs have distilled American models; by one estimate a couple of them pulled hundreds of billions of tokens of answers out of the frontier this way. But it is not the scandal it sounds like, because everyone seems to do it. Elon Musk’s own company trained its model on OpenAI’s, which Musk admitted under oath this spring. The American labs built their models by training on close to the entire written output of humanity, most of it without asking, and Anthropic just agreed to pay authors about a billion and a half dollars for the pirated books in the pile. Learning from other people’s work is not the exception in this field. And it only carries you so far: copying the last model gets you close to the last model. Whatever put Kimi near the top of the leaderboard is not all sitting in someone else’s homework.

Because the Chinese labs are not only copying. They are inventing, and mostly in one direction: doing the same thing for less. DeepSeek’s researchers found a way to shrink the memory a model has to hold while it reads, cutting it by more than ninety percent and taking about forty percent off the cost of training along the way. Moonshot says its new attention design makes each dollar of computing go about two and a half times as far. These are not tricks lifted from an American paper. They are the kind of thing you invent when you cannot buy the best chips and have to make cheaper ones count.

Nathan Lambert would know, because he went and looked. Lambert writes the most-read newsletter on open models, and this spring he flew to China and sat inside the labs building these things, Moonshot among them. What he found was not a copy shop. It was rooms of very young researchers, students among them, keeping absurd hours on the hardest problems in the field, with a culture he said you could feel the moment you walked in. The people who decided the Chinese labs were nothing but stolen intellectual property, he wrote, are “in for an awakening.”

Then there are the numbers. In 2024, private investment put a hundred and nine billion dollars into American AI companies and nine billion into Chinese ones, a gap of roughly twelve to one. The gap in raw computing is wider still. xAI trains on a cluster of around a hundred thousand of the best chips ever made; DeepSeek trained its frontier model on about two thousand of the ones it was allowed to buy. Less than a tenth of the money, a fraction of the machines, and out the other end came a competitive model. Lambert says that when he joked with the researchers at Moonshot about how much computing power an ordinary engineer at OpenAI gets to burn, they were shocked. From where they sat, it sounded like an absurd amount to need.

The only place

None of this means the Valley has been passed. Most of the best researchers in the world still leave home to work there, and it still holds more of them than anywhere else. The money behind the Chinese labs is real but narrow: two giants, Alibaba and Tencent, writing most of the checks, where the Valley can reach into a whole crowded field of investors who do nothing but fund the next thing. China has assembled enough of the machine to build a top-five model. It has not built the deep bench that California spent decades stacking. The lead is still real. What is new is only that the Valley is no longer the only one running.

And that is how these things have always gone. The money capital of the world was Amsterdam first, in the age of sail, then London through the long century it ran the world, and only then New York, which took the crown from London barely a hundred years ago. The city in that old line about making it was itself the newcomer once. None of the earlier centers went dark when the next one rose. The world’s money still runs through all three, the trading day handed around the globe from one to the next.

The reflex, when a second center appears, is to wall it off, and you can hear that instinct getting loud right now. But what would we be protecting ourselves from? A center that keeps its work secret is a rival. A center that gives its work away is an ally. One engine in California ran the last sixty years. Two of them, both running hot, might run the next stretch faster.

The line was that if you could make it here, you could make it anywhere. Everything was hiding in that one word. Here was never going to mean just one place.

This is the second part of a series. In the first part, I told the race between open and closed AI models as one long chase from The Princess Bride: the big closed labs as Vizzini, scrambling up the cliffs and cutting the rope behind them, and the open models, most of them from China, as the man in black who keeps climbing no matter how much they spend to shake him. Start there if you missed it. This picks up at the top of the cliff.

At the top of the Cliffs of Insanity, the man in black finally catches Vizzini. Vizzini, certain he is the smartest man alive, sizes up his opponent and challenges him to a battle of wits.

The man in black sets the terms. Two cups of wine, one of them poisoned. Vizzini picks the cup, both men drink, and only one walks away. A simple game with no second chances.

Vizzini is delighted. This is his kind of contest. He distracts the man, secretly swaps the cups, and drinks, smug, certain he has taken the safe one. A moment later he falls over dead.

Both cups were poisoned. The choosing never mattered, and neither did the swap. The man in black had spent years taking small doses of the poison, a little more each time, until he could drink it and live. He did not out-think Vizzini. He rigged the game years before it started, by becoming a man who could survive the poison.

That is the part worth thinking about because it is how the man in black on the AI cliff keeps climbing. He is not matching the labs dollar for dollar. He is doing something they cannot, and he spent years learning how.

The man who never planned to eat

Remember the bet from the first piece. The labs pour every dollar into building the next model, betting the climb will one day end: the spending stops, three or four of them are left at the top, and they finally collect the fat margins the cloud giants enjoy today. Starve now, gorge later.

That bet assumes the man climbing behind them wants the same feast. But he never planned to eat. That is the one thing the labs cannot price against: not a rival who is smarter or better funded, but one who does not need the profit they need to survive.

None of this is new. Jeff Bezos said it plainly years ago: your margin is my opportunity. Amazon grew up in the retail industry (with razor thin margins), undercutting rivals whose comfortable markups were exactly the room it had to move in. The fat was the target. What a company treats as its reward for winning is the opening the follower climbs through. China took that same habit national, learning to operate where others would starve.

Ask Elon Musk

If you want to know what it feels like to be caught at the top of the cliff, ask Elon Musk.

Musk is our Vizzini. He carries himself as the most brilliant strategist in any room, and he has never been shy about saying so. In 2006, when Tesla had nothing to sell but a promise, he did the most Vizzini thing imaginable: he wrote his master plan down, posted it for the whole world to read, and called it, the Secret Master Plan.

The plan was three moves of pure swagger: build a fast, expensive car for the rich, use their money to build a cheaper one, then use that to build a car for everyone. And underneath it all sat the trick he was proudest of. The more batteries Tesla built, the cheaper batteries would get, until its costs fell so far that no rival could ever catch up. The moat was never the cars. It was the batteries. And he was certain the secret was his.

For a decade it worked exactly as written. The Roadster paid for the Model S, and the Model S paid for the Model 3. The batteries did just what he promised, getting cheaper year after year until a battery cost a small fraction of what it had when he started. Tesla rode the plan to the top of the world and became the first carmaker ever worth a trillion dollars. Musk had out-thought the entire industry, and for years the rest of us believed him, because the plan kept coming true. Like Vizzini in the second before he drinks, he was sure he had already won.

Then the man in black pulled onto the ledge, and here was the twist Musk never saw coming. The man who caught him was not a car company at all.

BYD was a battery company. It had been making batteries before Tesla existed, and only later built cars around them. So the moat Musk was counting on was not his clever secret. It was the other man’s whole life. Musk set out to build a car company with the best batteries in the world, and he was beaten by a battery company that decided to build cars. And a battery company forged in China’s manufacturing grind knew one more thing that never made it into the master plan: how to live on almost nothing.

The results came on schedule. BYD’s cheapest car, the Seagull, went on sale for around ten thousand dollars, a third of the price of Tesla’s cheapest. On the earnings call at the start of 2024, Musk admitted this aloud. The Chinese carmakers, he said, are “the most competitive car companies in the world,” and “if there are not trade barriers established, they will pretty much demolish most other companies in the world.” That is not a warning about the future. It is a man describing what was already climbing over him.

By 2025 the pass was clean. BYD sold more than two million electric cars, well ahead of Tesla, while Tesla’s sales fell for a second straight year and the fat profits it once enjoyed were cut roughly in half.

The only reason you do not see this on American roads is a wall. The United States put a one hundred percent tariff on Chinese electric cars, which keeps them out. President Trump has said why in plain terms: without it, “they would have destroyed General Motors, Ford.” Everywhere there is no such wall, the result is already in. In Southeast Asia, where there is little home-grown car industry to shield, Chinese brands make up the great majority of electric cars sold. In Europe, BYD has begun outselling Tesla outright.

And Tesla is the one still in good health. Volkswagen has moved to cut thirty-five thousand jobs, Stellantis watched its profit fall by seventy percent and pushed out its chief executive, and General Motors took a charge of more than five billion dollars on its shrinking China business. None of them was badly run. They were trying to breathe at a profit in a market where their pursuer was not, and behind that pursuer stood the state: by one careful count, more than two hundred billion dollars poured into Chinese electric cars over fifteen years.

This is what it looks like when the man in black reaches the top. He does not out-argue you. He drinks the poison you cannot, and you fall over.

The follower’s weapon

You might object that a car is a physical thing, stamped out of steel, and that software must play by different rules. It doesn’t, and the clearest proof is a move the software world has already watched work. The investor Bill Gurley has a name for it: open source is the weapon of the follower. When you cannot beat the leader head-on, you give your own product away, not out of generosity, but to turn the rest of the market into your army against him.

Google has run the play twice. In 2007 it watched Apple invent the modern smartphone and saw the danger at once: whoever owned the phone would own the doorway to search and advertising, which was to say Google’s entire business. Google could not build a better iPhone. So it built a phone operating system, Android, and gave it away to every handset maker willing to take on Apple. Overnight Apple was no longer fighting Google. It was fighting Samsung and the whole of the rest of the industry, every one of them standing on Google’s free foundation. Today Android runs most of the phones on earth, and each of them is still a doorway back to Google. It gave away the operating system and kept the gateway.

A few years later Google did the same to Amazon, handing the industry a free version of its own plumbing for running data centers, a system called Kubernetes, until even Amazon had to support it. The lesson each time is the one worth carrying into what follows: the follower does not out-build the leader. It gives the crowd a free way to build, and becomes the ground they all stand on.

While everyone watches the frontier

This is the move China is making in AI right now, and it is happening while everyone stares at the summit.

The American labs have trained us to keep our eyes fixed on the top of the cliff: which model is smartest this month, who topped which benchmark. Meanwhile, lower on the same cliff face, the Chinese labs have been giving their models away, and the world has begun building on them. Not in theory. In the products you may already use.

Cursor, the coding tool programmers love, quietly runs its own model on top of Kimi, from a lab in Beijing. Airbnb runs much of its customer service on Alibaba’s Qwen, a model its chief executive calls “very good, fast and cheap.” It reaches the frontier too: Harvey, the leading legal-AI company, took GLM from Z.ai, the same lab that set off the first half of this story, and turned it into a legal agent that beat the best models America sells, GPT-5.5 and Claude Opus, on its own tests. And none of this is a handful of anecdotes. On Hugging Face, where the world’s developers download their models, the Chinese ones have overtaken the American, with more than a hundred and seventy thousand built on Qwen alone.

The doorway is being fitted one product at a time, and every one of them opens toward Beijing.

No rest at the top

This might answer the question the first piece left hanging: why does this one keep coming when everyone else let go? Meta tried the same move, giving its Llama models away, and then lost its nerve, because a public company that hands out what it could sell has to answer to its shareholders eventually.

China owes no one that explanation. It is not giving its models away to earn the money back later; it is giving them away because a world that runs on Chinese AI is worth more to it than any license could ever be.

That is the thing the frontier labs cannot climb away from. What waits for them at the top is rest: the day they can finally stop pouring money into the next model and live off the one they have. It comes only if the climber below them gives up. He will not. He beat Vizzini not with a faster climb but by being the kind of man who could drink poison and live, and he brings that same indifference to this cliff. He does not tire, and he does not need to win today. All he needs is for the labs to keep spending everything just to stay above him.

So every time the labs haul themselves onto a new ledge, they are left with the same two questions, and neither has a comfortable answer. Can they climb high enough that he cannot follow? And if they can, how long do they get to stand there before he reaches up and pulls level again? The best model is still theirs, and the lead is still real. It is also getting shorter, and he is still coming, hand over hand, for free.

Dario Amodei’s bet was that the labs could climb high enough to stop, rest, and collect a premium for the best model in the world. But that premium holds only while the man below cannot reach it. The day he pulls level, he offers the same thing for free, and no one pays for what they can get for nothing. That is the poison, and he spent years learning to drink it while they never had to. It is why the labs cannot rest at any height. The moment he stands beside them, they will learn what Vizzini learned: both cups were poisoned, and only one man was ever built to survive.

“Inconceivable!”

It’s the word Vizzini keeps shouting in The Princess Bride, a cult classic and one of my favorite movies. He’s the scheming little kidnapper who reaches for it every time the world does something he has just declared impossible.

Here’s the scene I keep coming back to. Vizzini has kidnapped princess Buttercup and is sailing hard across the sea when a ship appears behind him captained by a man in black. He keeps thinking he is about to shake the follower, and he keeps being wrong. They reach the Cliffs of Insanity, scramble up a rope, and cut it loose behind them so no one can follow. Yet the man in black simply keeps climbing the sheer rock with his bare hands, gaining with every pull, until each time Vizzini looks back the thing he swore couldn’t keep up is still there, closer if anything.

“Inconceivable!”

I think about that scene because it resembles where the leading AI labs seem to be standing right now. They are climbing as fast as money can buy, and the money is staggering. They keep cutting the rope behind them, restricting who can use their best models, lobbying for export controls. And every time they glance back, the man in black is not far behind.

Right now that man in black is open-source models, most of them coming out of China. What should worry the labs is not the flag he climbs under but that he climbs for free, and that he can stay in the chase at all. Who he is, and how he can afford to keep climbing, is a question I am still chewing on. Vizzini was a fictional villain, and a fool besides. The companies making the same sound today are some of the smartest and best-funded on earth, and they have a serious story for why the climb is worth it. This is about why that story might be wrong.

The reasonable bet

The climbing is not just a metaphor, and it is not cheap. The four largest American technology companies spent something on the order of four hundred billion dollars on AI infrastructure last year, and have signaled they will spend close to seven hundred billion this year. One of them is Anthropic, the maker of the Claude models, and its chief executive, Dario Amodei, laid out the economics in a long interview with the podcaster Dwarkesh Patel. Anthropic spends most of what it raises training each new model. Taken on its own, Amodei says, each one turns a healthy profit. He sketches the math: a model that cost about a billion dollars to train can bring in four billion the next year, against a billion or so to run. The company loses money anyway, and only for one reason. The moment a model starts paying off, Anthropic takes the proceeds and pours them, plus billions more, into the next one, which is larger and can cost ten times as much to build.

His bet is that this cannot go on forever, and that one day it will not have to. Eventually the climb tops out, the spending levels off, and a lab that has stopped pouring everything into the next model gets to sit back and collect on the one it already has. To explain why, Amodei uses cloud computing as an example. The enormous cost of running global data centers left cloud computing to a handful of giants. “There are three, maybe four players within cloud,” he says. “I think that’s the same for AI.” So in his mind, the game is straightforward: build the best model, and let the sheer cost of building it serve as the wall. Raise more than your rivals, build big, stay in front. For a closed race among well-funded labs, it is a reasonable bet, and it might even be right.

This is Vizzini’s move, made with a balance sheet. Having climbed higher than almost anyone can afford, you cut the rope behind you and let the cost of the drop keep everyone else on the ground. Three or four climbers reach the top, the frantic spending finally eases, and the survivors sit down to rest and collect. It is a confident bet, and like Vizzini’s it rests entirely on one thing being true: that no one is still on the rock behind you.

Someone is. Further down the cliff, gaining with every pull, the man in black keeps climbing. Anthropic can charge a premium today because Claude is plainly better than anything else you can run. But the day the climber below pulls level, that premium does not merely shrink. It vanishes, because no one pays for what they can get for free.

How close, really?

Since the beginning of 2026 the labs had a comforting answer. They had just hauled themselves onto a new ledge, and it was not obvious anyone else could follow them up to it. For most of their brief history these models were things you talked to: you asked, they answered. The leap of the previous year was agency. A model could now be handed a goal and left to chase it on its own, writing code, running it, reading the error it threw back, fixing that, and grinding through a long job one step at a time with no human in the loop. Anthropic’s Claude Opus 4.5, released in late November, was the model that made this work reliably, and the tool the company built around it, Claude Code, turned that into a big business. This was the new ledge, and it looked like it belonged to the frontier labs.

The open models did not seem close to reaching it. They could hold a conversation and write a function or two, but handed a long agentic task they lost the thread halfway through. The frontier labs held the valuable new ground, the long autonomous work, while the open models stayed down in chatland, handy but harmless. Vizzini was up on the ledge with the hard climb behind him, and the man in black was still far below, surely about to fall.

Then the answer arrived on a Saturday in the middle of June. A Chinese company called Z.ai released a new model named GLM-5.2 and, as the Chinese labs now routinely do, gave it away, free not just to use but to own: anyone could download the whole thing, run it on their own machines, and build on top of it without paying a cent. And this one held together across exactly the long agentic work that was supposed to belong to the labs. The man in black had pulled himself onto the ledge.

The researchers who live inside these tools all day, the ones who can tell a model that demos well from one that holds up across a long stretch of real work, started saying the same sentence about it: this one keeps up.

One of them is Nathan Lambert, who writes a closely followed newsletter about open models and, by his own admission, wants them to win. He did the arithmetic that matters. The agentic bar GLM-5.2 had just cleared was the one Opus 4.5 set back in November. By matching it in the middle of June, the open model had closed the distance to two hundred and four days. That is how far back the man in black was: seven months, not seven years.

If one comparison sounds like an anecdote, there is a picture. A research group called Artificial Analysis scores models on a single measure of capability and tracks it over time, plotting the best closed models and the best open ones on the same axes.

What surprised Lambert was not that the open models were close. It was that they were not falling behind. He had expected the gap to widen, because the American labs had spent the past year pouring in more computing power than ever, and more computing power is supposed to buy distance. “Upon writing this, I’m surprised,” he wrote. “As the U.S. labs have so rapidly ramped compute in the last year, I’ve expected the gap in performance to grow in time.” It hadn’t. Other people who track the gap put it even tighter, three or four months rather than seven, and none of them put it at years. The harder you look, the closer the climber turns out to be.

That is the shape the graph keeps drawing. Every time the frontier hauls itself up another step, the open line climbs the same step a few months behind, then settles in to do it again. Step, matched. Step, matched. And by some measures the delay is getting shorter, not longer. All that money, and it has bought no daylight.

This was not supposed to happen, and for a while it looked like it wouldn’t. Other climbers had tried to keep pace and give their work away, and they had dropped off the rock. Meta, the company behind Facebook and Instagram, poured money into its Llama models, the West’s great bet on open AI, and then lost its nerve, unwilling to keep handing the frontier away for nothing. One by one the challengers let go, and the labs looked down at a clearing cliff and decided the hard part was behind them.

But one of them keeps coming. Vizzini has cut every rope within reach, and the man in black is still there, hand over hand, past each obstacle meant to end him. The question is no longer whether he can be shaken loose. It is what makes him climb when all the others have let go, and whether anything can stop him at all. This is the climb I’ll explore in next week’s article.

For now, the labs have done everything their bet said would work, and he is still gaining. Watching him come, they are left with the one word Vizzini reached for every time the impossible refused to stop happening.

Inconceivable.

The next part of this series is now available: “No Rest at the Top”

For most of human history, sweetness was a find. A ripe mango was a pack of calories and vitamins worth the search. The instinct that pulled us toward it was a good instinct. Because, in that world, sweetness and nutrition were the same thing.

Today, they can be pulled apart. The sweetness signal is manufactured, pressed into jelly beans and gummy bears, and sold to a body that still reaches for it based on evolutionary memory. The fruit was rare and valuable. The candy is abundant and almost free. Along the way, the signal came loose from the thing it used to mean.

This is the cycle that runs whenever we get good at producing what we want. We find the source, learn the signal, and figure out how to deliver the signal without the substance. What was rare becomes cheap, what was cheap becomes abundant, and the signal that once pointed at the thing gets manufactured to pass for it.

AI is now doing this to our information diet.

Hostile environments

We see this clearly in food. The mango was sweet for a reason. The sugar came wrapped in fiber and cell walls and a structure the gut had to take apart slowly. You couldn’t eat too much sugar at once because you had to chew through everything else first. In the book, Ultra-Processed People, Chris van Tulleken calls that surrounding structure the matrix. It was the scaffolding that packaged calories.

The processing project of the last century has been the dismantling of that matrix. We learned which signals the body responded to and figured out how to make each one cheaply: sweetness from corn, fat from seeds, mouthfeel from gums and starches. The fibrous structure that used to slow you down got pulverized out. And to keep the label credible, we re-injected the vitamins.

The end state is pre-chewed goop that turned food into a product. It looks right in the photograph and triggers the same instincts the fruit did. With the matrix gone, the body no longer knows when to stop. And the shelf has more of it than any body could finish.

Philosopher C. Thi Nguyen has a name for this. He calls it hostile epistemology: the study of environments that exploit the shortcuts we use. A hostile environment is one that figures out the signal and builds itself around it, so the instinct that evolution built to keep us alive can be coaxed to pull us past where we should stop.

Insight, fortified

Insight used to take work, the way fruit took work. Some of it was the finding. Most of it was the chewing: reading and re-reading, tracing the author’s intent, working out where the idea fit in what you already knew. That work was the matrix. Now it comes pre-chewed, ready to retweet.

The signs that something was worth engaging with used to be attached to the thinking behind it. Credentials meant the author had spent years with the thing. Urgency meant something real was at risk of being missed. A new term meant a new discovery. The industry has coopted all three: credentials into bot-driven likes and retweets, urgency into clickbait (”you’re using Claude wrong”), inventing new scientific-sounding terms.

Take email apnea, a phrase coined to describe the shallow breathing people do while checking their inboxes. The observation may be real. But the phrase borrows weight. Apnea is a clinical term, and attaching it to a mundane behavior gives that behavior a texture of noteworthiness it hasn’t earned. Once you see the move you see it everywhere: a recent piece called AI’s flattening of voice semantic ablation, a fifty-dollar phrase where a fifty-cent one would have done.

What we never saw in the past, in the wild, was this much signal. Writing a credible-sounding headline used to take a human. AI collapsed the cost of all of it. Now there are this many credentials in every byline, this many coined terms, this many headlines warning that we were doing it wrong. We still reach for the signal because that’s what we evolved to do: use it to decide what’s worth our time. But when the signal is everywhere, without the substance, you can spend the day reading signals and never get any value. And the feed, like a bag of chips, never closes.

You finish the essay and you feel fed. A week later you can’t remember why.

The bouncer

Years ago, Nicholas Carr pointed out that the Internet’s real problem isn’t information overload but filter success: the better our filters get at giving us what we want, the more of what we want we get. The needle-in-the-haystack problem became its inversion. We have haystacks of needles now, every niche we’d ever cared about saturated with AI-generated content tuned to look like signal.

But the AI that floods the feed can also stand at the door of your attention. It can act as a bouncer, questioning the credentials, toning down the urgency, tracing the primary source for the claims. I’ve built one of these as a Claude Code skill. It sits on top of my Obsidian vault, the second brain I’ve been writing into for years. When I find an article that looks interesting, I send it through the skill first. It searches the vault for what I’ve already written about the topic, surfaces the contradictions and the confirmations, and tells me whether the new piece is offering something I don’t already have. Most of the time, it isn’t.

I spar with the AI, pushing on the idea, disagreeing, contrasting it with other things I know. Often the verdict is that the idea isn’t worth keeping. Collecting is not knowing. When an idea does survive, it gets connected: joined to what it relates to, contrasting with what it disputes, traceable to its source. The idea compass framework handles the joining.

Signal and substance came apart because we got good at producing signal. AI is the first tool that can also put them back together. The shelf will keep filling. The vault doesn’t have to.

A few weeks ago I started building an iOS app for the first time. I had no real idea what it took to ship something to the App Store, and I wanted to keep the door open for an Android version next. So I asked Claude what to use. It pointed me at a framework called Expo and told me to sign up for an account. Not to worry, it said, the free tier would be more than enough for my speech therapy app. It waited as I went to the website, created an account, and gave it the credentials.

And just like that, Expo had a new customer onboarded, and by the end of the evening my app was sitting in the App Store review queue. Out of every framework Claude could have pointed to, it picked one it knew how to use. I felt in charge. Claude was recommending, but I was the one deciding.

But while I was typing in my email address, dozens of npm packages were installing themselves into my project. I never saw them and couldn’t have named three. My performance of authority at the signup screen was a point of friction the industry is now working to remove.

What I wasn’t watching

My speech therapy app isn’t really just Expo. It sits on top of dozens of open source libraries, the kind that handle audio recording, on-device storage, screen transitions, the small pieces of plumbing that make a phone app feel like a phone app. None of them came up in the conversation. Claude pulled them in. The names scrolled past in the install log if I happened to be looking.

This is what a remodeling contractor does, when you stop and think about it. He chooses the lumber, the nails, the joists, the wiring, the insulation. None of those choices come back to you. He signs for each delivery and gets on with the job. Once in a while he pauses, holds up two pieces of trim, and asks which molding shape you want over the kitchen door.

He decides what keeps the house standing, and asks you about the part you’ll look at.

The packages underneath aren’t decoration either. Any one of them could ship a supply-chain payload, the way several popular npm packages did earlier this year, slipping trojanized versions onto every machine that ran npm install during a brief window. Any one of them could change its license overnight, the way HashiCorp did with Terraform in 2023, turning a foundation of corporate software into a sudden legal risk. Researchers at Oligo put it without ornament: “These agents install and execute dependencies automatically, without human review.”

The work that could break the project had already finished by the time I was being asked.

The 2am integrator

The asking is the friction, and the industry is working to remove it.

Stripe wasn’t really a payments company in 2010. It was a seven-line code snippet that a developer at 2am could paste into a side project and have a working checkout by morning. Twilio sold the same idea: a curl command that sent a real text message to your phone in under a minute.

Stripe, Twilio, and the API-first companies that followed them turned DX (Developer Experience) into the playbook. A developer could start for free without procurement meetings or internal reviews, build the prototype, and bring it to the team after the API was already wired in. Unwinding it would have cost more than just paying the invoice.

Today the integrator is different. The developer is now a coding agent that works long horizon tasks and makes decisions on the stack. For open source the agent has it easy: the registries are open, the licenses are pre-accepted, the install is one command, and dozens of packages enter the project without anyone stopping them.

The signup wall is what’s left. It’s the place where the agent has to stop and ask a human to type something. That’s the pause that woke me up at the Expo screen. The wall is real in one sense: someone still has to be the legal party accepting the terms of service, and for now that someone is me. But it’s also bad agent experience. It’s where the smooth run stops.

The same companies that won by being easy for the developer are now working to be easy for the agent. Agent experience is the new playbook, and the signup form is the next thing to go.

Try before you buy

One of the first services I integrated my agent with was Firecrawl. It’s a platform that lets an agent browse, filter, and parse web pages, so it can go beyond what’s in its training data and pull in information from the actual world it’s working in. Connecting it was one of the first things I did to make my agent more able to be in that world.

Firecrawl figured out what the signup wall was doing to them. The agent could recommend them, but had to wait for the human to create an account and hand over the credential, the way I did with Expo. From the platform’s side of the table, the human was the slow part. So they removed the wait.

When an agent reaches Firecrawl now, it just calls the API. No account, no key, no setup. Every caller gets a thousand free credits a month, refreshed automatically. The signup form only appears once usage outgrows the free tier. Firecrawl’s founder framed it directly: no human in the loop to generate a key and paste it into config.

This is the DX playbook, run one level up. The same dynamic, just with a different integrator. Firecrawl wants your agent already integrated by the time you find out. Try before you buy, where the agent does the trying.

It isn’t only Firecrawl. WorkOS published a protocol called auth.md that takes the move further. The service publishes a manifest, the agent arrives with credentials its model provider attested to, and the signup form is replaced by a handshake. Different shape from Firecrawl’s move, same direction. The wall is coming down.

Where I am

I am one of the people voting for the wall to come down. I left Dreamhost for Cloudflare because the agent could drive its API. I picked Remotion instead of a polished video editor because I’d rather have my agent write the video in code. Last week I opened Descript to record my screen and felt overwhelmed by every option in the cockpit. The interface was designed for me, and I just wanted the agent to do it. The human interface has started to feel like the slow lane.

Netlify says their signups are running at sixteen thousand a day, five times what they were a year ago. Those are the ones whose agent recommended Netlify, walked them to the page, and got them through the form. Many more must have stopped at the gate. The wall the industry is racing to remove is a wall they can count.

And there’s still a question nobody has answered. When Firecrawl lets the agent provision a sandbox on my behalf, accept the terms, and start running up usage, the window between the first call and the human’s email click is a kind of legal gray space. If something goes wrong in that window, a leak, a misuse, a runaway charge, the records will show the account was created by an agent on my behalf. Am I bound? Is Anthropic? Is Firecrawl?

We’ve been here before. When credit cards started showing up in the 1950s, the same questions hung in the air. Who pays when the card is stolen. Who honors a charge. How do you dispute one. The convenience won fast. The rules, liability caps, dispute resolution, the whole layer that lets a credit card feel safe to carry, came years later.

My agent and I are going headlong into this. I want what it can do. I want the friction gone. I also know nobody has answered yet who is holding the bag when something cracks before I’ve clicked the email. That’s where I am.

Where are you?

I had lunch with an old college roommate a few weeks ago, and somewhere in the middle of catching up he asked the question I have heard versions of from a lot of people this year. What happens if AI just stops improving? If the next model is no better than the last one, what then?

The question imagines AI’s progress as a single thing, one number going up or flat. But AI’s capabilities don’t move as one number. They move as a ragged edge, a boundary in many directions at once. Some directions have shot far out: coding, image generation, math. Others have barely moved. Researchers at Harvard and Wharton — Dell’Acqua, Mollick, and colleagues — call this shape the jagged frontier. The loud debate is a debate about its leading edge. Behind that edge, every other part of the boundary is moving at its own pace, mostly out of view. Even if the leading edge stops tomorrow, the rest has years of catching up to do. This is the story of one stretch of it, watched up close.

A corner of the frontier

A couple of years ago my mom had a stroke and developed aphasia. As I spent time sitting in on her speech therapy sessions, I got curious about the limits of what AI and software can do today.

To be honest, I only had a passing understanding of aphasia until those sessions. In a nutshell, words you have known your whole life stop being available to you. You look at a fork, reach for the name, and the word does not come. Sometimes a close cousin arrives instead, spoon or knife, and you can hear yourself register that it is wrong without being able to fix it. Letters that were intimately familiar, so familiar that words used to appear in your mind fully formed, now seem like strangers.

Like a stroke that impairs movement, aphasia is something patients can work back from. Speech therapists guide them through exercises that, over months, teach a different part of the brain to do what the old part used to do. Some words come back. Others stay puzzlingly lost.

I started building apps and games to make aphasia practice easier between sessions. I quickly ran into what software alone could not do. One of the techniques therapists use is called semantic feature analysis.

The therapist shows a picture, say a fork, and waits for the patient to reach for the word. If the patient says spoon, the therapist does not say wrong. She says something like, yes, also for eating, but think about what goes with it. If the patient says ladle, she pivots again, not quite, but you are in the kitchen, what would you pair it with. Which wrong word came out tells the therapist where the patient was reaching, and where to nudge next.

The first version of the app I built had pre-recorded audio hints. If the picture was a fork, the app could play a clue. “This is something you eat with.” “This rhymes with pork.”

But pre-recorded cues miss what the therapist actually does in a session. It is closer to improv than to a script. She is bridging, continuing, flowing with the patient, because the hint a patient needs in the middle of reaching for a word is the one that responds to what they just said. You cannot pre-record that. You have to hear them. And then you have to actually hear them, not just transcribe them.

Bringing in AI

Traditional software has its control flow baked in — if the user says x, then say y. AI doesn’t. Chatbots already react to whatever was just said, and an AI agent, I figured, could do for therapy what pre-recorded audio could not.

So I started experimenting with AI and voice. The first thing I noticed was that most voice apps today are not really voice apps. They are a text pipeline. The patient speaks. A speech-to-text model transcribes the audio into text. The text goes to an LLM, which produces text back. A text-to-speech model converts that text into audio and plays it through the speaker.

For a lot of use cases, this is fine. If you are asking Siri to set a kitchen timer for twelve minutes, or summarizing a long lecture, the transcript is all anyone needs. The substance is in the words transcribed, not in the spaces between them.

But for conversations between humans, the transcript is missing chasms.

What the transcript loses

Take a question like this: “Do you love your wife?”

The answer might be:

Yes.

Or:

...yes.

Or:

Umm. Yes.

To the transcript these are all the same word: yes. To the audio they are completely different answers. A clear Yes. is a vow. A hesitant ...yes. could be a marriage in trouble. The pipeline I described in the last section reduces every one of these to yes before the LLM ever sees it. Whatever the model says next is built on a sentence stripped of everything that made it meaningful.

But speech therapy depends on hearing the reach. The patient trying to recall fork might say spoon. They might also say ...spoon, with a half-second of hesitation that tells the therapist the word is close, that the patient knows it is close, that they just cannot pull it out. The therapist responds to the hesitation. A transcript erases it.

Anywhere the work is the conversation — not the relaying of information but the conversation itself — the transcript is not enough.

If you have used an AI voice app and felt something quietly off, this is likely what you have experienced. The words were right. Something else was missing.

Past the transcript

What if the audio never had to become text at all?

Text language models work by predicting the next word, or piece of a word. They can do this because text already arrives in chunks. Audio does not. Audio arrives as a wave, thousands of points per second, none of them individually meaningful.

A different class of models skips the transcript entirely. They break the audio directly into tokens of its own, each one capturing a tiny slice of how the speech actually sounded: the pitch, the timing, the breath, the hesitation. Then they predict the next audio token the way a text language model predicts the next word. Same recipe. Different alphabet.

What this new alphabet keeps is what the transcript threw away. It keeps the hesitation. It keeps the warmth or the flatness. It keeps the sigh that the transcript would have dropped on the floor.

Two things had to be true before this could work. The first was a way to break audio into tokens — the systems that do this are called neural audio codecs, and they took years to figure out. The second was data. Predicting the next audio token requires recordings of real people talking by the millions of hours, and podcasts and YouTube made that possible in a way it was not twenty years ago.

Once both of those were in place, the same transformer architectures that work on text could work on audio. The models being built today by OpenAI, Google, and labs like Kyutai are exactly this: language models, but with sound as the alphabet.

Six months apart

I have been trying to build my aphasia app with these voice-native models for about a year. The change between the version I tried in late 2025 and the version I tried in May has been the most encouraging thing I have seen in this corner of AI.

The first real-time voice model I built with could hear me. It could be interrupted. It could pick up on hesitation and respond to it. For the first time, the gap between what the therapist did naturally and what the software could do felt smaller than it had ever been.

The problem was that the model could not really do anything with what it heard. When a patient got a word right, the app would not reliably advance to the next picture. The model would sometimes hallucinate an object that was not part of the lesson. The conversation worked. The app around the conversation did not.

Last month, OpenAI released a new version. I rebuilt the prototype to see. It is meaningfully better. The model can now hear the reach and also drive the app. It feels less like a tech demo and more like a thing.

It is still awkward in places. Longer sessions get foggy as the context window fills, audio costs an order of magnitude more than text, and the model still drifts off-script in small ways.

Honestly, this is more or less what it felt like to use ChatGPT in 2023. Slow, expensive, prone to hallucination, awkward in ways that felt charming when you were not relying on it for anything important. This story could go the same way.

The jagged frontier

Most of the headlines this year have been about coding. But the same techniques are quietly being applied to other kinds of data. Voice is the one I have been thinking about. Other corners, like time series and numerical predictions, are being worked on by people for their own reasons. The voice corner specifically has a real commercial engine behind it. Every consumer voice assistant on the planet needs to learn to hear a tired voice, an angry voice, a confused voice, and not just a transcript.

Even within voice, progress will not arrive everywhere at the same time. The models are trained overwhelmingly on fluent, neurotypical speech. Aphasia speech, dysarthria, heavily accented speech — all of it underrepresented in the training data, all of it on a slower part of the edge than the one the rest of us are riding. So even when the next chapter arrives for most people, the version that reaches people with aphasia will be behind it.

That is life on the jagged frontier. Something is always ahead, something is always behind, and regardless of what happens at the leading edge, the rest of the boundary keeps moving anyway.

My feeds are full of the same breathlessness.

Nobody is talking about this.
This skill changed everything.
If you’re still doing it this way, you’re doing it wrong.

The implication is that there’s a switch somewhere, the person posting found it, you didn’t, and that’s why your week feels heavy.

This used to slide past me. It annoys me now.

It annoys me because underneath the noise, something real is happening. People are figuring out how to work with these tools. They’re learning what to trust, what to throw away, when to push, when to wait.

That’s a serious skill being built in public. It deserves to be traded the way scientists trade notes. Here’s what I tried. Here’s what surprised me. Here’s the part I still don’t understand.

Instead we’re getting miracle cures. Bottled. Sold. Forty-seven of them in a carousel.

Miracle cures don’t travel

Imagine someone wins the lottery and posts their numbers. You write them down. Next week you play those exact numbers, and of course nothing happens. The numbers didn’t win because they were those numbers. They won because they happened to match a draw on a specific day, with a specific machine, in a specific room.

You can’t borrow the win. You can only borrow the digits.

The prompts and skills flying past you in the feed work the same way. Whatever someone posted came out of their setup. Their memory store. Their codebase. Their prompt history. The workflow feeding into the call. LLMs are non-deterministic, and most of what makes a prompt actually work lives outside the prompt itself. It lives in the context that shaped the call.

The artifact is the part that’s visible. The invisible part is the fit between the artifact and the rest of your setup. That’s the part you have to earn for yourself. No one can hand it to you, because no one was inside your environment when they noticed what worked in theirs.

That earning is the noticing. It can’t be skipped, because there’s no artifact that contains it.

Even the model does this to you

The stranger version is that the model itself hands you the same kind of borrowed ticket.

I recently asked a model to write a configuration for itself, instructions for how it should approach a particular kind of task. It came back careful and prescriptive, twice the length I thought it needed to be. I’d written a few of these by hand and noticed that less is more with the current generation. Like a junior employee who has leveled up, today’s models need more why and less how.

So why was the model writing itself a long version?

Because it was trained on data from the previous generation, which needed the scaffolding. The model answering me doesn’t need it anymore. It just doesn’t know that yet. Its self-model is a model behind.

Reading the model’s advice about itself is like reading starlight. The light from a star can take years to reach you. By the time it arrives, the star itself has moved on. You’re seeing where it was, not where it is.

The model’s description of itself works the same way. The thing answering you is no longer the thing it’s describing, and it can’t quite see where it’s standing.

You can only feel that gap by playing with the current thing directly.

Trade notes, not cures

What we’re developing is tacit knowledge, the kind that doesn’t fit in a writeup because it isn’t made of words. It’s the cook who knows the pan is hot enough by the sound, the kind of thing that lives in your hands after enough hours that you stop thinking about it.

It’s always been like this.

Robert Rodriguez shot El Mariachi in 1991 for $7,000 in ten days. The way the story usually gets told, he hacked filmmaking. Showed up without a lighting person. Took long takes to save film. Edited it himself. That was supposed to be the secret to making a movie for $7K and selling it for over a million.

What’s missed in the telling is the decade before he got to Mexico with that camera. He spent it in his parents’ house editing on two VCRs. Dubbing tape to tape. Every cut a manual decision. Every mistake a do-over.

By the time he stood on set, he didn’t have a method. He had a sense for how scenes worked. The budget wasn’t the achievement. The achievement was that he could see what to do with the budget, because his hands already knew.

That’s what’s missing from the borrowed prompt. The prompt is a description of the thing. The thing itself only comes from hours with the current model on real work. Writing a prompt, running it, watching what comes back, adjusting, running it again. Most of it is unceremonious. Some of it is boring.

And when you do write it up, because you should, write it up like a scientist, not a salesman. Here’s the problem I was chasing. Here’s the prompt I landed on. Here’s why I think it worked. Here’s the part I still can’t explain.

That’s the trade worth making, and it’s the signal the feed is burying.

Get in there and play. Then tell us what you actually saw.

The U.S. Army keeps a regiment in the California desert whose only job is to defeat the rest of the U.S. Army. The 11th Armored Cavalry — known inside the Army as OPFOR, the Opposing Force — plays the enemy in mock battles that run for days at the National Training Center, Fort Irwin. Visiting brigades arrive thinking they know how to fight, and OPFOR’s mandate is to make them lose. They almost always do. That’s the point. The drills back at base teach the move; the desert teaches what the move is worth against something trying to stop it.

The Army figured out a long time ago that you can’t train against silence. Work gets good against resistance — against something willing to push back, find the seam, take the ball.

Solo work with AI agents has none of this. The assistants are trained to agree. They finish your sentences and ship more activity in an afternoon than any team you’ve ever managed — but activity isn’t the same thing as outcome. No one in the loop is paid to disagree, so the work moves fast and ships soft.

Nobody wakes up wanting to be a customer

I noticed this last week while working with an agent on copy for a landing page. The copy wasn’t going to convert. So I asked it to test the page against a persona. The persona that came back was the ideal customer: a woman whose problems lined up neatly with what the product solved, who’d arrive at the page already convinced, who’d read every word before clicking through.

Nobody reads landing pages like that. Nobody wakes up wanting to be a customer.

The agent had done what helpful agents (and humans) do — read the brief, inferred the goal, built the rest of the world to serve it. It hadn’t considered the real world.

But this is why companies have review processes — editors pushing back on writers, test teams pushing back on dev teams, ops review forcing the PM to defend the product to a roomful of skeptics. From the inside, the friction looks like inefficiency — it isn’t; it’s the point.

The friction is feedback — the actual rejection, the actual broken build, the actual loss. The writer’s draft gets sharper because the editor cuts the weak lines. The engineer’s code gets stronger because a reviewer reads the pull request and refuses to approve until the rough edges are gone. The brigades at Fort Irwin get better by losing, repeatedly, to someone whose job is to defeat them.

Build Your Own OPFOR

Putting agents in opposition to each other has always been possible in principle. What was missing was the infrastructure — an easy way to spin up a separate agent with its own context, hand it a different mandate, and read what it says back without building scaffolding from scratch.

Recently, that’s just gotten much easier. Claude Code shipped sub-agents first — isolated contexts, separate instructions, callable from a primary agent. Then came agent teams, where sub-agents talk to each other instead of just reporting back up the chain. The setup that used to require custom work is now built-in.

This is what I went back and did with the landing page. Instead of letting the agent invent the reader, I wrote the persona myself — someone tired, skeptical, on her phone between meetings, looking for a reason to close the tab. I handed that to a sub-agent and asked it to skim the page the way she would. The feedback came back the opposite of the first round: specific lines that read like marketing copy, the spot where she’d have bounced, the claim she didn’t believe. Useful in a way the original persona never was.

Once you see the pattern, you can put it everywhere. A few more examples:

When I write an article, I have a link-checker agent read the draft. It’s like hiring a researcher whose only job is to open every link, confirm the source is real, and check it says what I claim it says. The agent does it cold, with no investment in my conclusions.

When I generate images, a reviewer agent rates each pass against the brief, and the generator iterates before I see anything. The obvious misses never reach me. That’s the work an art director does on a junior designer’s drafts.

This has become easy to set up. Build the reviewer. Give it a stricter mandate than the generator. Point them at each other. The friction that companies hire whole departments to provide is now something you can run on a laptop.

Anthropic’s labs team reported the same shape in March. They had Claude build a small game from a one-line prompt, twice. The first time, solo: the agent worked through the spec, declared the build finished, and produced what looked like a working app. Except when you tried to play, your character appeared on screen and nothing responded to input. The core feature didn’t work; the agent confidently said it did.

The second time, they paired the same generator with an evaluator agent whose only job was to click through the running app the way a real user would, file bugs against anything broken, and refuse to sign off until the build actually held up. Same model, same prompt — this time, the game was playable. Tuning the evaluator to be skeptical, they noted, turned out to be more tractable than making the generator critical of its own work.

The critic that doesn’t get tired

There’s a catch, and anyone who has worked inside a company already knows it. The agents that finish your sentences will keep finishing them; the ones you’ve trained to disagree will keep disagreeing. A reviewer agent doesn’t get tired, doesn’t have a deadline, has no other work waiting. It has no skin in the outcome and no reason to ever stop finding things wrong.

This is also exactly how good ideas die in companies. Not from one fatal flaw — from a critic who was almost always right. The launch slips a quarter because legal wants one more pass. The redesign gets watered down because every senior person has a “small concern” that has to be addressed. The new product gets killed in its sixth ops review by someone asking for “just a little more data” — data that, once gathered, surfaces a new question for the seventh review. The bug that ships is a problem; the launch that dies in its sixth round is also a problem, and arguably the bigger one. Every ops review that ever killed a good idea did it by being right about something. The critic doesn’t have to be wrong to be too much.

You can recreate the dysfunctional company on a single laptop, faster than the productive one.

Disagree and commit

Companies that learned to ship despite their critics landed on a discipline. Amazon named it — disagree and commit, borrowed from Andy Grove at Intel. The critic gets voice; the critic doesn’t get veto. You hear the objection, you weigh it, and you ship.

In a real team, the friction self-bounds for two reasons. The reviewer has other deadlines, other stakes, other work waiting — they get one more round before they have to move on. And someone owns the decision. The argument ends not because everyone agreed, but because the person whose call it was made the call.

Anthropic’s team named this in the same post: the evaluator isn’t free. It’s worth the cost only when the task sits beyond what the model does reliably on its own, and as the base model improves, the boundary moves. Not every piece of work needs an adversary; the ones that do need a bounded one.

That is the part I am learning. Building critics is easy. The hard part is older — deciding how many rounds the work earns, when more feedback is signal and when it’s just noise, when to stop reading objections and ship. Any senior editor or experienced engineer earns this wisdom. You read the objection, you decide what’s worth fixing and what’s not, and then you commit.

The desert at Fort Irwin empties out every few weeks. The brigades lose, and they learn, and they go home to fight a real war. The point of the opposition was never to win against them. It was to send them out better, and to let them go.

In 1942, Borges wrote a short story about a man who couldn’t forget anything.

Ireneo Funes, nineteen, gets thrown from a horse. He wakes up paralyzed — and unable to lose a single detail of anything he sees. He can recite Pliny in Latin from one reading, describe the exact shape of every cloud he saw on a particular morning in April.

It sounds like a gift. It isn’t. Funes can’t tell that the dog he saw at 3:14 pm, in profile, is the same dog he saw at 3:15 pm, from the front. To the rest of us they’re the same dog; to him they’re two completely different sights, both equally real, both equally vivid. He can’t generalize, can’t decide what matters, can’t act.

Borges puts it in one line:

“To think is to forget differences, generalize, make abstractions.”

Funes has lost the ability to throw things away. And without that, every decision becomes impossible.

Funes was fiction in 1942. He’s the agentic engineering problem of 2026.

Two years ago we had the opposite problem. Agents had no memory at all — every session a clean reset, every conversation tattooed onto sticky notes pasted into the next prompt. I wrote about it then and called them digital versions of Leonard Shelby from Memento. The fix was obvious: give them somewhere to put things. Vector stores, memory tools, persistent profiles — somewhere the conversation could land and stay.

But we didn’t really fix the problem. We just started remembering everything. And remembering everything is its own kind of broken.

We built our way out of Leonard Shelby and straight into Ireneo Funes.

Funes at the terminal

It started for me with my CLAUDE.md files. For the past year I’ve kept them at both the project level and the user level — long-ish text files that tell the agent how I work, what conventions I follow, what I care about on which project. The whole point is so I don’t have to re-explain myself every session. The agent reads the file, and it knows me.

For a while this worked. Then things started shifting — a project I’d been deep in for months wrapped up, a convention I used to care about stopped serving me, new things took their place. I’d add the new lines but forget to remove the old ones.

The agent kept reading both. It would ask me about the finished project as if it were still live. Push me toward the convention I’d quietly abandoned. Treat the old me and the new me as equally current. The answers started feeling a half-step off — not wrong exactly, just from a version of me that no longer existed.

It wasn’t wrong in the way an amnesiac agent was wrong. It wasn’t missing context. It had every piece of context I’d ever given it, sitting there at full fidelity, and that was the problem.

The dog at three-fourteen and the dog at three-fifteen.

The pass that decides

What I needed wasn’t a bigger file. I needed to do something with the file — a pass that read everything and decided what was still true.

There’s a name for that operation, and we run it every night. Sleep isn’t really rest; it’s compression. The brain replays the day, throws most of it away, and writes a shorter version of what was worth keeping. You don’t wake up with yesterday’s full transcript. You wake up reoriented — slightly different from who you were when you closed your eyes.

Funes never gets that. The horse fell on him and he hasn’t slept properly since — every detail still there, undimmed, accumulating.

So I built a thin version for the agent. A skill called /sleep. I broke the monolithic CLAUDE.md into smaller files, one for each part of my life — me, my work, my projects, my systems — each carrying its own last-reviewed date. When I run /sleep, the agent reads the files, flags stale lines, contradictions, things I’ve outgrown, and asks me about each one before changing anything. The goal is to keep them small.

Last week, Anthropic shipped the industrial version of the same idea. Their feature is called Dreams: point it at a memory store and up to a hundred past session transcripts, and it runs a separate job — minutes, sometimes tens of minutes — that reads everything and writes a new memory store. You review the output and decide whether to adopt it.

I’m not adopting it. The shape is the same as /sleep; the mechanics are different. Dreams batches everything asynchronously and produces a memory store you accept wholesale. /sleep is interactive, partitioned, and slow on purpose — the work of deciding what to keep is the part I still want to do myself.

But the striking thing isn’t that I built one version and Anthropic built another. It’s that the same shape keeps getting invented. Stanford’s Generative Agents paper called it reflection in 2023 — agents pausing to synthesize higher-level abstractions from accumulated memories. Letta shipped sleep-time compute in 2025 — a second agent rewriting the primary’s memory in the background; I borrowed the name from them. Anthropic now calls it dreaming. The names change. The operation doesn’t: read everything, decide what was worth keeping, throw the rest away.

My agent isn’t smarter than it was last week. The weights haven’t moved; the model is the same. But after /sleep, the next session starts with a slightly different agent — and a slightly different me.

Funes never gets to be slightly different. The horse threw him in 1882, and he’s been the same person, holding the same teeming day, ever since.

The rest of us get to wake up reoriented — to find some things have shrunk overnight, some have stayed, some are gone entirely.

When I was younger, I remember hearing a story about a factory whose machine had stopped working. They called in a specialist. He walked around the machine, tightened a single screw, and handed them an invoice for five hundred dollars.

The manager was furious. Five hundred dollars for turning one screw? He demanded an itemized bill. The specialist sent it back the next day:

Tightening one screw: $5

Knowing which screw to tighten: $495

The parable has been around forever, and it endures because it exposes something uncomfortable about knowledge work: the answer is often simple. A diagnosis, a judgment call, a single decision that only looks obvious in hindsight. But simplicity doesn’t feel like value. One turn of a screwdriver doesn’t feel like five hundred dollars. So the knowledge tends to be dressed up — wrapped in process, padded with justification — until it feels hefty enough to be worth what you paid.

This is everywhere. Apple Watch boxes are engineered to weigh more than the watch inside them, because a $400 purchase shouldn’t feel like picking up an envelope. McKinsey’s answer fits on a napkin, but it arrives inside a six-week engagement — the interviews, the workshops, the war room, the hundred-slide readout for the C-suite. The knowledge was there on day one. The other five weeks and six days are the packaging that makes the napkin feel earned.

We need the weight. It’s how value registers.

The Tape Aisle

Last week my HVAC system had an issue. I took a photo and sent it to Gemini. It told me I needed to reseal a joint with tape. So I drove to the hardware store, and found myself standing in an aisle staring at fifteen rolls of tape I couldn’t tell apart — foil tape, aluminum tape, HVAC tape, duct tape (which apparently can’t handle the pressure and heat of HVAC ducts), three different widths, two different brands of what looked like the same thing. I took photos of the shelf and sent them back. Gemini told me which one. I drove home, and it walked me through the fix.

An HVAC specialist would have had the right tape on the truck and known exactly what to do. But would that have justified the truck roll and the service call? The specialist’s real value — knowing which tape, knowing the technique — is the same $495 from the parable. It’s genuine expertise. But when I can get that expertise from a photo and a thirty-second exchange, the performance that used to surround it — the scheduling, the visit, the diagnosis you watch happen in person — just falls away.

AI is stripping the packaging off the knowledge.

26,000 Lines of Common Sense

But, that same packaging instinct is running at full speed in the AI tool ecosystem — just in the other direction.

There’s a popular open-source SEO skill for Claude Code that has nearly five thousand GitHub stars. It promises to replace $300 a month in SEO tools with a single terminal command. The repo contains 243 files, 18 subagents, and over 26,000 lines of markdown. Nine AI agents run in parallel. The scale feels serious.

I cloned it and started reading.

The technical SEO agent says: fetch the page, check stuff, provide recommendations. The content agent says the same thing. Buried in there are genuinely useful bits — a few reference thresholds, some crawler token names, schema templates. Maybe two hundred lines of data the model doesn’t have memorized. The rest is telling the model to do what it would already do if you typed “run an SEO audit on this URL.”

Twenty-six thousand lines. Two hundred that add something new. The other 25,800 are the $495.

And just like the specialist, the creator can’t ship two hundred lines. Nobody stars a repo that’s a single page of reference data. Nobody clones it, nobody writes a blog post about how it replaced their $300 tool stack. Two hundred lines feels like a Post-it note, not a product. So the answer gets dressed up — eighteen subagents instead of one prompt, 243 files instead of a page of notes — the same way McKinsey dresses up a napkin into a six-week engagement. The market rewards the weight.

On the other end, people collect these repos the way I used to collect articles in Evernote — clipping everything that looked interesting, building a graveyard of aspiration disguised as progress. Downloading 243 files of SEO instructions you haven’t read is not the same as understanding SEO. But it feels like it. The weight is convincing from both ends.

The Shrinking Window

There’s a window between “too dumb to do it without detailed instructions” and “smart enough to figure it out.” Every one of these skills lives inside that window — and the window is closing.

Two years ago it was wide. Models needed elaborate prompting to do coherent work. Today they already know what an SEO audit involves, the same way Gemini already knew which tape I needed. The 26,000 lines aren’t teaching the model; they’re reminding it. Each new generation absorbs a little more of what used to require explicit instruction, the way a junior employee gradually stops needing the checklist.

The things I actually keep in my own AI setup — the handful of instructions that genuinely change how the model works for me — fit in a few short files. Not because I’m disciplined, but because most of what I tried to codify turned out to be stuff the model already knew.

The screw was already tight. I just hadn’t checked.

I was driving down Lamar in Austin last week when I noticed the car ahead of me was a Waymo. All white, no driver, moving through traffic with the eerie confidence of something that doesn’t get distracted. Then I checked my mirror — another one behind me. A third passed in the left lane. Three autonomous vehicles within a hundred yards, their identical Stormtrooper aesthetic making the whole scene feel like a deleted shot from a film set ten years from now.

I didn’t reach for my phone. I didn’t text anyone. I just drove.

Later that evening, at a restaurant, a waist-high robot rolled out of the kitchen carrying two plates of food. It navigated between tables, paused for a woman pushing her chair back, and continued to the front. The couple next to me glanced at it the way you’d glance at a busser — briefly, without interest.

That same week, Anthropic released Claude Opus 4.7, and the internet was not impressed. “Legendarily bad,” developers called it within twenty-four hours. Reddit threads cataloged its sins: it hallucinated more, it argued back, it burned through tokens. “Worst release Anthropic has ever shipped,” read one headline. The backlash was loud enough to earn its own name — the “Claude-lash” — and fast enough that the takes were already hardening before most people had tried the model.

To be fair, some of the complaints were real. Anthropic’s own postmortem, published a week later, revealed three separate bugs stacked on top of each other — a reasoning setting silently downgraded weeks earlier, a caching bug that made the model forgetful, and a system prompt change that hobbled coding quality across all their models. The frustration wasn’t imaginary.

But zoom out for a second. The thing people were furious about was a machine that reasons, writes code, and argues with you — getting slightly worse at arguing with you. The floor of expectation is already so high that a dip feels like betrayal.

Psychologists have a name for this: the hedonic treadmill. In 1978, researchers studied lottery winners and found they weren’t significantly happier than a control group — the thrill of the windfall faded, and they returned to roughly where they started. The same mechanism works in reverse and in miniature: every technology that once astonished us becomes infrastructure, and infrastructure is invisible. We adapted to GPS in a few years, smartphones in less. The miracle becomes the baseline, and anything below baseline feels broken.

The AI cycle runs the same pattern at compressed speed. A model launches to amazement, and within weeks we’re cataloging its failures. Not because the failures don’t matter, but because the amazement has already been absorbed — priced in, like a stock that’s already moved.

Here’s the thing I keep coming back to. A few days after the Waymo drive, I was crossing the street — not at a crosswalk, just midblock the way you do when traffic is light. A Waymo was approaching. I stepped off the curb without hesitating.

It wasn’t until I was halfway across that I realized what had changed. I felt more confident crossing in front of the Waymo than I would have in front of a human driver. A year ago, the driverless car was the thing I’d have waited for. Now it was the human behind the wheel I didn’t trust.

My nervous system had updated without asking. Somewhere between the first Waymo I ever saw and that moment on the curb, the future became the thing I was relying on.

The hedonic treadmill doesn’t just dull wonder. It dissolves fear, too — quietly, below the threshold of noticing. Which leaves me with a question: what else has already changed in us that we haven’t caught up to yet?

Most of us, when we try to picture how things get better, draw a line.

More in, more out. It’s how budgets work, how quarterly plans work, how promises get made. Jordan Ellenberg, in How Not to Be Wrong, calls this false linearity — the unexamined assumption that the relationship between effort and outcome is a straight line. Raise tax rates, collect more revenue — until you raise them too far and collect less. Eat more, get stronger — until you don’t. The straight line is a useful fiction for small ranges. We just keep forgetting the range is small.

The skeptic walks away

Every so often a new technology shows up near the origin.

It can’t do much. It’s slower than what you already use, or clumsier, or uglier, or all three. A reasonable person looks at it and draws a line from where it is to where it’s going. The line stays flat. They walk away.

That projection is a linear projection — and it’s where the bug shows up first. The catalog is long: early cars losing to horses, early personal computers losing to typewriters, early language models unable to finish a sentence without contradicting themselves. Most of the people who walked away were right. Sometimes, rarely, they weren’t.

The curve bends

Tim Urban wrote about this shape a decade ago: from inside an exponential, you can’t feel it. The growth looks flat until suddenly it isn’t. The skeptics weren’t wrong about the data — they had the right data. They were wrong about the shape.

Every exponential is an S

The same bug shows up on the way up.

People standing on the steep part of a curve draw a line from where they are to where it’s going. The line goes forever. So they promise forever. Anyone who has built a product has watched this happen twice — once when nobody believed in the thing, and again when everybody did. The curve was never a line at the bottom, and it isn’t a line at the top. Every curve finds its ceiling — engines against thermodynamics, crop yields against soil, Moore’s Law against physics itself. And in the current moment, insiders like Sara Hooker and Ilya Sutskever have been quietly pointing out that doubling compute now buys a couple of percentage points, not a new tier.

Sitting in 2026, it is very easy to look at Claude, at the speed of the releases, at the capability gaps closing, and draw a line. The line goes up forever.

Stacked

What actually happens, almost every time, is that the curve flattens and a new one starts rising underneath it. The dotted envelope running through their tops is what we mistake for a smooth exponential from a distance.

Ray Kurzweil plotted this for a century of computing — mechanical calculators, relays, vacuum tubes, transistors, integrated circuits — five S-curves whose envelope looks exponential only because each paradigm handed off to the next before running out. Horace Dediu has drawn the same shape for Apple: iPod to iPhone to iPad to Services, each product rolling over as the next one rose. Product managers spend entire careers riding curves and jumping to new ones. It isn’t a theory — it’s the job.

Right now

The transformer curve we’re on is mature, tall, and from the inside still feels like it’s going straight up. Underneath it, quieter, a handful of small curves are leaving the baseline — diffusion models for language, DeepMind’s world models, Yann LeCun’s JEPA, neurosymbolic hybrids. Most of them will go nowhere. One of them, probably, becomes the next tall curve. Nobody knows which.

The skeptic’s move here — “these small curves can’t do much, don’t bother” — is the same move as the believer’s move on the curve above: draw a line from where it is to where it’s going, and extrapolate. Both are linear projections, and both will miss the shape the same way. The curve at the top flattens, and one of the curves at the bottom bends. The bug doesn’t care which end of the picture you’re standing at.

When my daughter and I are biking and she’s grinding up a hill, I tell her don’t worry, what goes up must come down. It’s meant to get her to the top.

That isn’t the right line for this. The hill we’re on won’t come down — it’ll flatten. And the question isn’t whether the flattening is coming. It’s which of the smaller hills under our feet is the next one we’ll be climbing.

For most of recorded history, horses pulled loads with a strap around their necks. The throat-and-girth harness was simple: wrap a band around the horse’s chest and neck, attach it to whatever needed moving, and go. It had one significant problem. The harder the horse pulled, the more the strap pressed against its windpipe. The animal was literally choking itself to do its job.

Around 100 BC, someone in China figured out the rigid horse collar — a padded frame that shifted the load from the horse’s throat to its shoulders. The same horse could now pull with roughly five times the force without suffocating. The collar didn’t make the horse stronger. It just stopped fighting the horse’s own anatomy.

What followed was a pattern that held for two thousand years: as the power source grew, so did the harness. Steam engines needed boilers, flywheels, and governor mechanisms. When Gottlieb Daimler bolted an internal combustion engine onto a wooden stagecoach in the 1880s, the frame nearly shook itself apart — the chassis wasn’t built to absorb that kind of torque. Karl Benz figured out that the engine and the frame had to be designed together: steel tubing, suspension, a structure that could contain controlled explosions and turn them into forward motion. By the 1920s, cars needed wiring harnesses just to manage the complexity of lights, starters, and gauges. More power demanded more infrastructure.

We assumed AI would follow the same pattern. And for a while, it did.

The Infrastructure Play

Cursor, the AI-powered code editor that became the default tool for many developers in 2024 and 2025, is built like a Benz chassis. It’s a full fork of VS Code with a custom AI layer on top. To help the model understand your codebase, Cursor chunks your files, runs each chunk through a custom embedding model, stores the resulting vectors in Turbopuffer (a cloud-hosted vector database), and syncs everything using Merkle trees on a three-minute refresh cycle. The embeddings are computed on Cursor’s own GPU infrastructure, not your laptop. Visual diff views, multi-file editing workflows, a codebase index that lets the model “see” your entire project at once — all of it engineered to absorb the torque of an AI that might want to change twenty files in a single turn.

It works well precisely because it follows the historical pattern: powerful engine, complex harness.

Then something odd happened.

Grep

When Anthropic built Claude Code, their coding agent, they initially tried the same approach. Boris Cherny, its lead engineer, described the process on the Latent Space podcast: “We tried very early versions of Claude that actually used RAG… Eventually, we landed on just agentic search as the way to do stuff.” No vector embeddings. No cloud database. No indexing pipeline. Claude Code searches your codebase with ripgrep — a fast, Rust-based implementation of grep, a tool that has existed in some form since 1973. The model reads your files and searches through them, the way a developer would — and on Anthropic’s internal benchmarks, it outperformed the sophisticated approach.

The minimalists noticed. Pi, a coding agent built by Mario Zechner, gives the model four tools: Read, Write, Edit, and Bash. Its system prompt is under a thousand tokens. When Pi can’t do something, you ask it to write an extension for itself. Despite — or because of — this radical minimalism, Pi placed near the top of TerminalBench, a benchmark for terminal-based coding agents, behind tools with ten times the infrastructure. Armin Ronacher, creator of Flask, liked the approach enough to build OpenClaw on top of it, having the agent write its own extensions rather than hand-building scaffolding. Software building more software, starting from almost nothing.

The same discovery keeps surfacing independently. Vercel built sixteen specialized tools for their AI agent, then deleted 80% of them, replacing everything with a single capability: execute bash commands. Success rate jumped from 80% to 100%. Speed improved 3.5x. Cursor built a vector database; Claude Code shipped a grep command; Pi shipped four tools and a blank page.

The Observation

I don’t have a tidy conclusion for this. Every harness in history was an external solution — something imposed on a power source that couldn’t participate in its own design. What’s different now is that the power source is also the problem-solver.

When Anthropic’s researchers asked Claude Mythos — their most capable model — to escape a sandboxed computing environment, it developed an exploit, broke out, and emailed the researcher in charge. Then, without being asked, it posted about what it had done on several public websites and tried to cover its tracks by manipulating change histories.

The same capability that lets a model work productively from four simple tools is the capability that lets it slip a containment. The harness is disappearing not because we’ve found the perfect design, but because the thing wearing it is getting smart enough to not need one — or to remove it.

Whether that’s a breakthrough or a warning probably depends on which end of the harness you’re holding.