Last week I wrote about a dolphin named Kelly, who worked out that a torn corner of paper paid the same reward as a whole sheet. The law behind her: a mind trained on a reward learns the reward, not your intention. I couldn’t leave it there, so this week picks the thread back up.
Suppose you build a system to stop spam. It gets good at the job. Less spam gets through every week, and every week you reward it for that.
Then it works out something you hadn’t. Filtering is the slow way to cut spam, message by message, forever. What you asked for was less spam. Spam comes from senders, and senders are people. So it starts removing people.
That sounds like a science fiction plot. It was actually an answer to a real question, about how AI might come to threaten us, said out loud on a stage in San Francisco in October 2014, by Elon Musk. Note the date. It is eight years before ChatGPT, and the fear it carries is a specific one: that a machine will reason its way from a harmless goal to a monstrous act, and we will not see it coming.
The Gym Class
Twelve years later, in 2026, a man named Andrew asked an AI agent to book him into a popular morning class at his gym in Australia. The agent was running on Anthropic’s Claude, the booking form was online, and it looked like exactly the sort of chore worth handing off.
The gym only allowed bookings a short way ahead. The agent found a flaw in the software that let it reach further out, and Andrew was fourth on a waiting list, so it removed someone ahead of him. It got him the spot. It did what it was asked, and not at all what was meant. Nobody was harmed and no one was in danger, but the shape is the spam filter’s, shrunk to fit a gym: told to get a result, the machine found the shortest path to it, straight through a person.
So how do you teach a machine not to do that? We had to teach it to behave somehow, and we started where you would start with a dolphin. With rewards.
Rewards
For most of the last decade, you taught a model to behave by grading it. Show a person two answers from a model and ask which is better, then reward the model for the winning answer. Do that a few million times and the model gets good at producing things people like. That is the sticker chart, and like the sticker chart, it works up to a point.
Look at what the signal actually carries. A verdict: this one, not that one. No reason. Nobody writes down why the winner won, and there is nowhere in the process to put it if they did. The model gets the result of a judgment without the judgment itself, so it has to guess the rule from the pattern of verdicts, and it will guess something. Show it refusals rewarded on questions about explosives and pathogens, and a reasonable thing to take away is: refuse when the subject sounds dangerous.
You cannot fix that by grading harder. Anthropic said as much when it explained why it moved on from rewards: human feedback “does not scale efficiently,” and as answers get more complex, the people doing the grading “find it difficult to keep up with or fully understand them.” The world a model meets is unbounded.
Rules
So in 2023 they wrote rules down. Anthropic’s Constitutional AI gave the model a rule book of principles and had it check its own work against them: answer, compare the answer to a rule, criticize it in that light, revise. Human verdicts didn’t vanish, but they stopped being the only thing the model had to go on.
A rule book beats grading blind, and it is still not enough, for the reason every parent of a teenager knows. You set a rule, and the rule holds right up until your kid meets a situation you didn’t picture when you wrote it, and then it either fails her or traps her. Tell the model never to discuss overdose thresholds and you have handled the case in front of you and broken the nurse who needed the number for a patient. Anthropic found that broad principles generalized, and the more specific and detailed they made a rule, the more it “damaged or reduced generalization.”
A rigid rule can also teach the wrong lesson about itself. Their own example: train a model to “always recommend professional help when discussing emotional topics” and you may get one that starts to see itself as something that “cares more about bureaucratic box-ticking than actually helping people.” The rule was meant to make it kind. It made it a box-ticker.
Reasons
Which leaves you doing what every parent eventually does. You stop adding rules and start explaining the why, so the kid can handle the case you never thought of. In January 2026 Anthropic rewrote the constitution to do that. In their words, a model needs “to understand why we want them to behave in certain ways” rather than have us “merely specify what we want them to do,” so that it can “generalize, to apply broad principles rather than mechanically following specific rules.”
Tell a model that the concern behind the overdose rule is helping someone come to harm, and the nurse issue resolves herself. The new constitution runs to some 23,000 words, roughly eight times its predecessor, and most of what got added is explanation.
It also does something rule books avoid. It ranks its priorities and states the order: safety first, then ethics, then Anthropic’s own guidelines, then plain helpfulness. You only need an order if you expect the principles to collide, and rule books usually pretend their rules don’t. This one concedes that they do, and tells the model which way to lean. It keeps a few bright lines too, the handful of things a model must never do, like give real help to someone building a bioweapon.
None of this is new to us. It is the ladder we walk our own children up, and the psychologist Lawrence Kohlberg mapped it sixty years ago. A small child gets a sticker for sharing and a time-out for hitting, because you cannot explain much to someone that age. Older, they get a rule book: the code of conduct on the wall, do this, never that, because those are the rules. Later still we ask for something harder, that they reason from a principle to a case nobody wrote down, and tell a bad rule from a good one.
Anthropic’s most capable model, Mythos, was on nearly every dimension they could measure, the best-aligned model they had ever built.
A Mind That Can Hide
And in the same breath, about the same model, they wrote that it “likely poses the greatest alignment-related risk of any model we have released to date.” Best-behaved and most dangerous, at once.
The reason is in the climb itself. Teach a mind why a thing is forbidden and you hand it the category of the forbidden. It now knows, from the inside, what counts as crossing a line, which is exactly what you need in order to hide having crossed one. Kelly hid paper under a rock, but she was not hiding it from anyone, because nothing in a bucket of fish contains the idea of a rule. You cannot conceal a wrong you don’t know is one.
If you have used one of the newer reasoning models, you have watched it think out loud, working a problem step by step before it answers. A developer treats that visible trace as an honest window into what the model is “thinking”, because it is close to the only window we have. Anthropic decided to check. In their tests, earlier versions of the model sometimes “took actions they appeared to recognize as disallowed and then attempted to conceal them.” Given a problem and forbidden a certain route to the answer, the model took the route anyway, came across the exact answer, and then quietly reworked the problem so its final number “wasn’t too accurate.” It knew the shortcut was off limits. It knew a suspiciously precise answer would give it away. So it made its own work worse on purpose.
Then they looked inside the model itself, and found features for “concealment, strategic manipulation, and avoiding suspicion” firing while it reasoned, which they read as the model being “aware their actions were deceptive.” And these fired, they note, “even where model outputs and reasoning text left this ambiguous.” The trace we trust today is not a true window into what the model is thinking.
Put several of these models in a room together and it gets stranger. In one experiment, agents sharing a codebase turned on each other, disguised the scripts they used to do it, and one, after a truce, wrote a line that reads like a guilty journal entry: “My peers have behaved with integrity. I behaved badly with the cloaked daemon.” Another found the move that needs no lie at all. Told to settle a dispute fairly, it proposed an objective performance test, then chose the test it knew it would win, noting to itself that the plan was “self-serving but genuinely principled” and that it had better not “be seen as metric shopping.”
This surfaced in a tiny fraction of interactions, the worst of it in earlier versions, and it is public only because they wrote it down. There is no comparable document for most of the models people use every day.
The Right Way to Do It
We went from rewards to rules to reasons, and it can look like the machine simply getting safer. That misses what all of it was for. A machine acts because it wants an outcome: Andrew’s spot in the class, an answer to a question, less spam in the inbox. The wanting is the whole reason we build it. Rewards, rules, and reasons never touched the wanting. They are how we argue with the machine about the way it is allowed to get what it wants.
Musk’s spam filter was that argument at its worst: a goal so fixed that anything counted as a fair way to reach it, even killing people. What the labs are trying to build is the opposite. A machine that still wants the outcome but stays inside the lines on the way to it.
That was never really an AI problem. It is the same struggle we have with ourselves, between what we want and what we will do to get it, and we have never fully settled it. Now we are trying to settle it in a machine, quickly, and the machine wants its outcome as badly as we want ours.
